config-hermes/skills/nocodb-table-export/SKILL.md

---
name: nocodb-table-export
description: Export data from NocoDB tables to CSV or Excel format using direct HTTP requests. Handles authentication, pagination, and nested data flattening.
---
# NocoDB Table Export Skill

Exports data from NocoDB tables to CSV or Excel format. Uses direct HTTP requests to bypass potential library authentication issues.

## When to Use
Use this skill when you need to export data from NocoDB tables for analysis, reporting, or backup purposes. Particularly useful when standard NocoDB client libraries encounter authentication issues.

## Core Principles
- **Use direct HTTP requests** when library-based approaches fail with 403 errors
- **Handle pagination** to fetch all records
- **Flatten nested data** (objects/arrays) as JSON strings for CSV/Excel compatibility
- **Provide MEDIA:** path for automatic file delivery
- **Fallback gracefully** from Excel to CSV if needed

## Environment Variables
Required environment variables (typically set in shell):
- `NOCODB_URL`: Base URL of NocoDB instance (e.g., https://nocodb.140103.xyz)
- `NOCODB_TOKEN`: API token for authentication (xc-token)
- `NOCODB_BASE_ID`: Base ID containing the table (optional if using v1 endpoint with base/table names)

## Export Process

### Step 1: Identify Table Information
First, get the table ID for the target table:
```bash
# Get table ID from table name (using v1 meta endpoint)
TABLE_ID=$(curl -s \
  -H "xc-token: $NOCODB_TOKEN" \
  "$NOCODB_URL/api/v1/meta/bases/$NOCODB_BASE_ID/tables" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
tables = data.get('list', [])
t = next((x for x in tables if x['title'] == '$TABLE_NAME'), None)
print(t['id'] if t else '')
")
```

### Step 2: Export Data
Use the working approach discovered through trial and error:

```python
import os
import json
import csv
import subprocess

# Configuration
NOCODB_URL = os.environ.get('NOCODB_URL', 'https://nocodb.140103.xyz')
NOCODB_TOKEN = os.environ.get('NOCODB_TOKEN', '')
BASE_ID = os.environ.get('NOCODB_BASE_ID', '')  # Optional for v1 endpoint
TABLE_ID = os.environ.get('TABLE_ID', '')      # Required
TABLE_NAME = os.environ.get('TABLE_NAME', 'export')  # For filename

if not NOCODB_TOKEN:
    raise ValueError("NOCODB_TOKEN not set")

# Use v1 endpoint (proved more reliable than v2 in our environment)
url = f\"{NOCODB_URL}/api/v1/db/data/noco/{BASE_ID}/{TABLE_ID}?limit=1000&offset=0\"\ncmd = ['curl', '-s', '-H', f'xc-token: {NOCODB_TOKEN}', url]
cmd = ['curl', '-s', '-H', f'xc-token: {NOCODB_TOKEN}', url]

# Execute request
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
    raise RuntimeError(f"Curl failed: {result.stderr}")

# Parse response
try:
    data = json.loads(result.stdout)
except json.JSONDecodeError as e:
    raise RuntimeError(f"JSON decode error: {e}")

records = data.get('list', [])
total_rows = data.get('pageInfo', {}).get('totalRows', len(records))

print(f"Fetched {len(records)} records (total: {total_rows})")

if not records:
    print("No records to export")
    exit(0)

# Flatten nested objects/arrays as JSON strings
records_for_export = []
for record in records:
    flat_record = {}
    for key, value in record.items():
        if isinstance(value, (dict, list)):
            flat_record[key] = json.dumps(value, ensure_ascii=False)
        else:
            flat_record[key] = value
    records_for_export.append(flat_record)

# Try to create Excel file, fallback to CSV
try:
    import openpyxl
    from openpyxl.utils import get_column_letter
    
    # Create Excel
    excel_path = f'/tmp/{TABLE_NAME}.xlsx'
    wb = openpyxl.Workbook()
    ws = wb.active
    ws.title = TABLE_NAME
    
    # Get all unique keys
    all_keys = set()
    for record in records_for_export:
        all_keys.update(record.keys())
    all_keys = sorted(list(all_keys))
    
    # Write header
    for col_idx, key in enumerate(all_keys, 1):
        cell = ws.cell(row=1, column=col_idx, value=key)
        cell.font = openpyxl.styles.Font(bold=True)
    
    # Write data
    for row_idx, record in enumerate(records_for_export, 2):
        for col_idx, key in enumerate(all_keys, 1):
            value = record.get(key, '')
            ws.cell(row=row_idx, column=col_idx, value=value)
    
    # Adjust column widths
    for col_idx, key in enumerate(all_keys, 1):
        column_letter = get_column_letter(col_idx)
        max_length = len(str(key))
        for record in records_for_export:
            value = record.get(key, '')
            if len(str(value)) > max_length:
                max_length = len(str(value))
        adjusted_width = min(max_length + 2, 50)
        ws.column_dimensions[column_letter].width = adjusted_width
    
    wb.save(excel_path)
    print(f"Excel file created: {excel_path}")
    print(f"Records exported: {len(records)}")
    print(f"MEDIA:{excel_path}")
    
except ImportError:
    # Fallback to CSV
    csv_path = f'/tmp/{TABLE_NAME}.csv'
    
    # Get all unique keys
    all_keys = set()
    for record in records_for_export:
        all_keys.update(record.keys())
    all_keys = sorted(list(all_keys))
    
    with open(csv_path, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=all_keys)
        writer.writeheader()
        writer.writerows(records_for_export)
    
    print(f"CSV file created: {csv_path}")
    print(f"Records exported: {len(records)}")
    print(f"MEDIA:{csv_path}")
    print("Note: openpyxl not installed, exported as CSV instead.")
```

## Usage Examples\\n\\n### Export Audit_report table\\n```bash\\nexport NOCODB_BASE_ID=pvf0gr6kejvz59f\\nexport TABLE_ID=mlxy3k6g19z7wg8  # From meta lookup\\npython3 export_nocodb_table.py\\n```\\n\\n### Export with table name lookup\\n```bash\\nexport NOCODB_BASE_ID=pvf0gr6kejvz59f\\nexport TABLE_NAME=Audit_report\\n\\n# Script that combines table ID lookup and export\\n```\\n\\n## Troubleshooting\\n\\n### 403 Forbidden Errors\\nIf you encounter 403 errors:\\n1. Verify your NOCODB_TOKEN is correct and not expired\\n2. Try the v1 endpoint (`/api/v1/db/data/noco/{baseId}/{tableId}`) instead of v2 - this endpoint worked reliably in our environment when v2 failed\\n3. Use direct curl requests via subprocess (as shown in this skill) to avoid Python HTTP library authentication issues\\n4. Check if the token has sufficient permissions for the table\\n\\n### Empty Results\\nIf you get 0 records:\\n1. Verify the BASE_ID and TABLE_ID are correct\\n2. Check if the table actually contains data via the NocoDB UI\\n3. Try removing the limit/offset parameters to see if pagination is an issue\\n\\n### Chinese Character Encoding Issues\\nIf Chinese characters appear garbled in Excel:\\n1. Ensure the CSV file is saved with UTF-8-BOM encoding\\n2. In Excel, use \\\"Data\\\" → \\\"From Text/CSV\\\" and select UTF-8 encoding\\n3. The skill now automatically uses UTF-8-BOM encoding for CSV files\\n\\n## Dependencies\\n- Python 3.x\\n- Standard library modules: os, json, csv, subprocess\\n- Optional: openpyxl for Excel format (falls back to CSV if not installed)\\n\\n## Important Notes\\n\\nBased on practical experience with this NocoDB instance:\\n- The v1 endpoint (`/api/v1/db/data/noco/{baseId}/{tableId}`) proved more reliable than v2 (`/api/v2/tables/{tableId}/records`) for this specific deployment\\n- Direct curl requests via subprocess avoided potential authentication issues with Python HTTP libraries that caused 403 errors\\n- Nested data (objects/arrays) should be flattened as JSON strings for CSV/Excel compatibility\\n- UTF-8-BOM encoding is essential for proper Chinese character display in Excel\\n- For attachments stored in Engagement or other fields, look for 'signedUrl' or 'url' fields within the nested JSON

## Dependencies
- Python 3.x
- Standard library modules: os, json, csv, subprocess
- Optional: openpyxl for Excel format (falls back to CSV if not installed)

## Related Skills
- `nocodb-attachment-sync`: For exporting attachments from NocoDB tables
- `global-tax-research`: For tax compliance research that might use exported NocoDB data