estuary/agent-skills / skills/materialize-generic-create
materialize-generic-create
UnverifiedCreate a materialization for any Estuary destination connector. Use this generic skill when no dedicated skill exists for the target system. Use when user says "materialize to", "send data to", "destination connector", or "set up materialization".
Create Materialization (Any Connector)
Create a materialization using flowctl for any supported Estuary destination connector. This generic skill handles connectors that don't have a dedicated skill.
Check for a Dedicated Skill First
Before using this generic workflow, check if a dedicated materialize-<connector>-create skill exists for the user's target destination. Dedicated skills have connector-specific guidance and troubleshooting that this generic skill can't provide. Only use this skill if no dedicated one exists.
Step 0: Find the Connector and Load Documentation
Find the connector
Query the connector registry to find available materialization connectors:
# List all materialization connectors
flowctl raw get --table connector_tags \
--query 'protocol=eq.materialization' \
--query 'select=image_tag,documentation_url' \
--output yaml
To search for a specific connector by name:
flowctl raw get --table connector_tags \
--query 'documentation_url=ilike.*<connector-name>*' \
--query 'protocol=eq.materialization' \
--query 'select=image_tag,documentation_url' \
--output yaml
Load the documentation
Use WebFetch to load the documentation_url returned from the query. This is the canonical docs page with prerequisites, config reference, and setup instructions.
Search Kapa for tribal knowledge (if the Estuary MCP is configured):
Search kapa ai knowledge sources for "materialize <connector-name> common issues"
If Kapa MCP is not configured, the user can set it up: https://docs.estuary.dev/features/mcp-integration/
Step 1: Gather Requirements
Before writing any YAML, ask the user:
- Target system details? — Connection info (host, port, credentials, etc.)
- Network path? — Direct connection, SSH tunnel, or other networking requirements
- Non-default data plane? — Most users use the default. Ask if they need a non-default data plane.
- Source collections? — Which Estuary collections to materialize
- Any connector-specific requirements? — Check the docs loaded in Step 0
- Hard deletes? — Off by default. Without it, deleted rows stay in the destination marked with
_meta/op: 'd'. Enable to physically remove them. - Delta updates? — Off by default. Switches from one-row-per-key (standard merge) to append-only. Use for event logs or history tables.
- Sync schedule? — Controls how often batches are written to the destination (default: 30 minutes,
0sfor real-time). Affects latency and destination compute cost.
Step 2: Find the Correct Connector Version
Use the image_tag from the Step 0 query. If you need to re-query for a specific connector:
flowctl raw get --table connector_tags \
--query 'documentation_url=eq.<DOCS-URL-FROM-STEP-0>' \
--query 'select=image_tag,documentation_url' \
--output yaml
Use the returned image_tag — never hardcode a version.
Step 3: Help User Complete Prerequisites
Walk the user through prerequisites from the docs loaded in Step 0. Common prerequisites across materializations:
- Destination accessible — Network connectivity from Estuary cloud
- Credentials configured — User/service account with write permissions
- Target location exists — Database, schema, bucket, etc.
Refer to the docs page for connector-specific prerequisite details.
Step 4: Create the Spec File
Build flow.yaml using the config reference from the docs. General structure:
materializations:
<TENANT>/<PATH>/materialize-<connector>:
endpoint:
connector:
image: <IMAGE>:<VERSION>
config:
# Properties from the docs config reference
# Each connector has different required fields
bindings:
- source: <TENANT>/<collection-path>
resource:
table: "<TABLE_NAME>"
# Some connectors use different resource fields
# (e.g., "topic" for Kafka, "index" for Elasticsearch)
Fill in the config section using the property reference from the connector's docs page.
For SSH tunnel (if supported by the connector), add networkTunnel.sshForwarding block — see docs.
Step 5: Publish
flowctl catalog publish --source flow.yaml --auto-approve
Step 6: Verify
# Check status
flowctl catalog status <TENANT>/<PATH>/materialize-<connector>
# View logs
flowctl logs --task <TENANT>/<PATH>/materialize-<connector> --since 5m | jq -c '{ts, message}'
Status progression:
PENDING— Normal for ~30 seconds during shard assignmentBACKFILLING— Initial data sync from collectionsOK— Running normally with real-time updates
Troubleshooting
Materialization stuck in PENDING
Wait 30-60 seconds — this is normal during shard assignment. If still stuck:
flowctl logs --task <TENANT>/<PATH>/materialize-<connector> --since 5m | jq 'select(.level == "error")'
Config validation errors
Cause: Config structure doesn't match the connector's schema
Fix: Re-read the docs page from Step 0 and verify all required properties are set with correct types and formatting.
"collection not found"
Cause: Source collection doesn't exist or wrong prefix
Fix:
flowctl catalog list --prefix <TENANT>/ | grep collection
Verify the collection name matches exactly.
"no connector tag found for image"
Cause: Wrong image name or tag
Fix: Re-query connector_tags (Step 0) for the correct image and version.
Network connectivity failures
Cause: Destination not reachable from Estuary cloud
Fix:
- Check firewall rules / security groups
- Verify Estuary IP addresses are allowlisted (see docs)
- Consider SSH tunnel for private networks
- For local testing: use ngrok or similar
Connector-specific errors
For errors specific to the connector, search Kapa:
Search kapa ai knowledge sources for "materialize <connector-name> <error message>"
Related Skills
schema-field-selection— Control which fields are materializedconnector-disable-enable— Pause/restart existing materializationsconnector-delete-recreate— Nuclear option for stuck materializationsestuary-logs— Deep log analysisestuary-catalog-status— Status checking
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |