aiskillstore/marketplace / skills/89jobrien/cocoindex
cocoindex
VerifiedComprehensive toolkit for developing with the CocoIndex library. Use when users need to create data transformation pipelines (flows), write custom functions, or operate flows via CLI or API. Covers building ETL workflows for AI data processing, including embedding documents into vector databases, building knowledge graphs, creating search indexes, or processing data streams with incremental updates.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 25.2 KB |
| references/api-operations.md | markdown | 13.3 KB |
| references/cli-operations.md | markdown | 8.8 KB |
| references/custom-functions.md | markdown | 13.5 KB |
| references/flow-patterns.md | markdown | 13.7 KB |
| skill-report.json | config | 173.3 KB |
references/cli-operations.md
View on GitHub---
author: Joseph OBrien
status: unpublished
updated: '2025-12-23'
version: 1.0.1
tag: skill
type: reference
parent: cocoindex
---
# CLI Operations Reference
Complete guide for operating CocoIndex flows using the CLI.
## Overview
The CocoIndex CLI (`cocoindex` command) provides tools for managing and inspecting flows. Most commands require an `APP_TARGET` argument specifying where flow definitions are located.
## Environment Setup
### Environment Variables
Create a `.env` file in the project directory:
```bash
# Database connection (required)
COCOINDEX_DATABASE_URL=postgresql://user:password@localhost/cocoindex_db
# Optional: App namespace for organizing flows
COCOINDEX_APP_NAMESPACE=dev
# Optional: Global concurrency limits
COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS=50
COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES=524288000 # 500MB
# Optional: LLM API keys (if using LLM functions)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
VOYAGE_API_KEY=pa-...
```
### Loading Environment Files
```bash
# Default: loads .env from current directory
cocoindex <command> ...
# Specify custom env file
cocoindex --env-file path/to/.env <command> ...
# Specify app directory
cocoindex --app-dir /path/to/project <command> ...
```
## APP_TARGET Format
The `APP_TARGET` tells the CLI where flow definitions are located:
### Python Module
```bash
# Load from module name
cocoindex update main
# Load from package module
cocoindex update my_package.flows
```
### Python File
```bash
# Load from file path
cocoindex update main.py
# Load from nested file
cocoindex update path/to/flows.py
```
### Specific Flow
```bash
# Target specific flow in module
cocoindex update main:MyFlowName
# Target specific flow in file
cocoindex update path/to/flows.py:MyFlowName
```
## Core Commands
### setup - Initialize Flow Resources
Create all persistent backends needed by flows (database tables, collections, etc.).
```bash
# Setup all flows
cocoindex setup main.py
# Setup specific flow
cocoindex setup main.py:MyFlow
```
**What it does:**
- Creates internal storage tables in Postgres
- Creates target resources (database tables, vector collections, graph structures)
- Updates schemas if flow definition changed
- No-op if already set up and no changes needed
**When to use:**
- First time running a flow
- After modifying flow structure (new fields, new targets)
- After dropping flows to recreate resources
### update - Build/Update Target Data
Run transformations and update target data based on current source data.
```bash
# One-time update
cocoindex update main.py
# One-time update with setup
cocoindex update --setup main.py
# One-time update specific flow
cocoindex update main.py:TextEmbedding
# Force reexport even if no changes
cocoindex update --reexport main.py
```
**What it does:**
- Reads source data
- Applies transformations
- Updates target databases
- Uses incremental processing (only processes changed data)
**Options:**
- `--setup` - Run setup first if needed
- `--reexport` - Reexport all data even if unchanged (useful after data loss)
### update -L - Live Update Mode
Continuously monitor source changes and update targets.
```bash
# Live update mode
cocoindex update main.py -L
# Live update with setup
cocoindex update --setup main.py -L
# Live update with reexport on initial update
cocoindex update --reexport main.py -L
```
**What it does:**
- Performs initial one-time update
- Continuously monitors source changes
- Automatically processes updates
- Runs until aborted (Ctrl-C)
**Requires:**
- At least one source with change capture enabled:
- `refresh_interval` parameter on source
- Source-specific change capture (Postgres notifications, S3 events, etc.)
**Example with refresh interval:**
```python
data_scope["documents"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="documents"),
refresh_interval=datetime.timedelta(minutes=1) # Check every minute
)
```
### drop - Remove Flow Resources
Remove all persistent backends owned by flows.
```bash
# Drop all flows
cocoindex drop main.py
# Drop specific flow
cocoindex drop main.py:MyFlow
```
**What it does:**
- Drops internal storage tables
- Drops target resources (tables, collections, graphs)
- Cleans up all persistent data
**Warning:** This is destructive and cannot be undone!
### show - Inspect Flow Definition
Display flow structure and statistics.
```bash
# Show flow structure
cocoindex show main.py:MyFlow
# Show all flows
cocoindex show main.py
```
**What it shows:**
- Flow name and structure
- Sources configured
- Transformations defined
- Targets and their schemas
- Current statistics (if flow is set up)
### evaluate - Test Flow Without Updating
Run transformations and dump results to files without updating targets.
```bash
# Evaluate flow
cocoindex evaluate main.py:MyFlow
# Specify output directory
cocoindex evaluate main.py:MyFlow --output-dir ./eval_results
# Disable cache
cocoindex evaluate main.py:MyFlow --no-cache
```
**What it does:**
- Runs transformations
- Saves results to files (JSON, CSV, etc.)
- Does NOT update targets
- Uses existing cache by default
**When to use:**
- Testing flow logic before running full update
- Debugging transformation issues
- Inspecting intermediate data
- Validating output format
**Options:**
- `--output-dir PATH` - Directory for output files (default: `eval_{flow_name}_{timestamp}`)
- `--no-cache` - Disable reading from cache (still doesn't write to cache)
## Complete Workflow Examples
### First-Time Setup and Indexing
```bash
# 1. Setup flow resources
cocoindex setup main.py
# 2. Run initial indexing
cocoindex update main.py
# 3. Verify results
cocoindex show main.py
```
### Development Workflow
```bash
# 1. Test with evaluate (no side effects)
cocoindex evaluate main.py:MyFlow --output-dir ./test_output
# 2. If looks good, setup and update
cocoindex update --setup main.py:MyFlow
# 3. Check results
cocoindex show main.py:MyFlow
```
### Production Live Updates
```bash
# Run with live updates and auto-setup
cocoindex update --setup main.py -L
```
### Rebuild After Changes
```bash
# Drop old resources
cocoindex drop main.py
# Setup with new definition
cocoindex setup main.py
# Reindex everything
cocoindex update --reexport main.py
```
### Multiple Flows
```bash
# Setup all flows
cocoindex setup main.py
# Update specific flows
cocoindex update main.py:CodeEmbedding
cocoindex update main.py:DocumentEmbedding
# Show all flows
cocoindex show main.py
```
## Common Issues and Solutions
### Issue: "Flow not found"
**Problem:** CLI can't find the flow definition.
**Solutions:**
```bash
# Make sure APP_TARGET is correct
cocoindex show main.py # Should list flows
# Use --app-dir if not in project root
cocoindex --app-dir /path/to/project show main.py
# Check flow name is correct
cocoindex show main.py:CorrectFlowName
```
### Issue: "Database connection failed"
**Problem:** Can't connect to Postgres.
**Solutions:**
```bash
# Check .env file exists
cat .env | grep COCOINDEX_DATABASE_URL
# Test connection
psql $COCOINDEX_DATABASE_URL
# Use --env-file if .env is elsewhere
cocoindex --env-file /path/to/.env update main.py
```
### Issue: "Schema mismatch"
**Problem:** Flow definition changed but resources not updated.
**Solution:**
```bash
# Re-run setup to update schemas
cocoindex setup main.py
# Then update data
cocoindex update main.py
```
### Issue: "Live update exits immediately"
**Problem:** No change capture mechanisms enabled.
**Solution:**
Add refresh_interval or use source-specific change capture:
```python
data_scope["docs"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="docs"),
refresh_interval=datetime.timedelta(seconds=30) # Add this
)
```
## Advanced Options
### Global Options
```bash
# Show version
cocoindex --version
# Show help
cocoindex --help
cocoindex update --help
# Specify app directory
cocoindex --app-dir /custom/path update main
# Custom env file
cocoindex --env-file prod.env update main
```
### Performance Tuning
Set environment variables for concurrency:
```bash
# In .env file
COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS=100
COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES=1073741824 # 1GB
```
Or per-source in code:
```python
data_scope["docs"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="docs"),
max_inflight_rows=50,
max_inflight_bytes=500*1024*1024 # 500MB
)
```
## Best Practices
1. **Use evaluate before update** - Test flow logic without side effects
2. **Always setup before first update** - Or use `--setup` flag
3. **Use live updates in production** - Keeps targets always fresh
4. **Set app namespace** - Organize flows across environments (dev/staging/prod)
5. **Monitor with show** - Regularly check flow statistics
6. **Version control .env.example** - Document required environment variables
7. **Use specific flow targets** - For selective updates: `main.py:FlowName`
8. **Setup after definition changes** - Ensures schemas match flow definition
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |