cianfhoghlaim/cianchosaint / .agents/skills/dlt
dlt
Verified| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 20.6 KB |
| references/crawl4ai-dlt-summary.md | markdown | 1.8 KB |
| references/dagster-dlt-assets.md | markdown | 3.9 KB |
| references/deploy-gcp-cloud-function-webhook.md | markdown | 2.1 KB |
| references/deploy-gcp-cloud-function.md | markdown | 2.0 KB |
| references/deploy-gcp-cloud-run.md | markdown | 2.4 KB |
| references/deploy-modal.md | markdown | 2.6 KB |
| references/destinations-cognee-memgraph.md | markdown | 3.3 KB |
| references/destinations-graphiti.md | markdown | 3.0 KB |
| references/destinations-lancedb.md | markdown | 3.4 KB |
| references/dlt-baml-orpc-mcp-typesafe-pipeline-analysis.md | markdown | 20.0 KB |
| references/dlt-transformations.md | markdown | 3.3 KB |
| references/dlthub-codebase-analysis.md | markdown | 32.4 KB |
| references/dlthub.md | markdown | 12.5 KB |
| references/openapi-generator.md | markdown | 2.3 KB |
| references/performance-optimisation.md | markdown | 3.3 KB |
| references/sqlmesh-init.md | markdown | 1.7 KB |
| references/type-safe-pipeline.md | markdown | 4.4 KB |
references/deploy-gcp-cloud-run.md
View on GitHub# Deploy: GCP Cloud Run (long-running)
Deploy a dlt pipeline as a **long-running** GCP Cloud Run job.
Useful for pipelines that take > 9 minutes (the Cloud Functions
limit).
## Pattern (Job)
```yaml
# job.yaml
apiVersion: run.googleapis.com/v1
kind: Job
metadata:
name: dlt-pipeline-job
spec:
template:
spec:
template:
spec:
containers:
- image: gcr.io/PROJECT/dlt-pipeline:latest
resources:
limits:
memory: 4Gi
cpu: 2
env:
- name: DLT_DISABLE_PLUGINS
value: "true"
- name: DESTINATION__BIGQUERY__CREDENTIALS
valueFrom:
secretKeyRef:
name: gcp-sa-key
key: latest
restartPolicy: Never
maxRetries: 5
completionMode: Indexed
```
## Deploy
```bash
# Build and push the image
gcloud builds submit --tag gcr.io/PROJECT/dlt-pipeline:latest
# Run the job
gcloud run jobs execute dlt-pipeline-job --region europe-west1
```
## Schedule with Cloud Scheduler
```bash
gcloud scheduler jobs create http dlt-pipeline-schedule \
--schedule "0 2 * * *" \
--time-zone "Europe/Dublin" \
--http-method POST \
--uri https://europe-west1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT/jobs/dlt-pipeline-job:run \
--oauth-service-account-email your-sa@PROJECT.iam.gserviceaccount.com
```
## Pattern (Service)
For a long-running service that polls a queue or watches a file
system, use a Cloud Run service (not a job):
```yaml
# service.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: dlt-pipeline-service
spec:
template:
spec:
containers:
- image: gcr.io/PROJECT/dlt-pipeline:latest
ports:
- containerPort: 8080
env:
- name: DLT_DISABLE_PLUGINS
value: "true"
```
## Cianfhoghlaim usage
The Cianfhoghlaim stack uses **Dagster on `bunchloch` (M4 Mac) + `arm1-oci`
(ARM)** for orchestration. GCP Cloud Run is a valid alternative
for cloud-only deployments.
## Reference
- The `Deploy with Google Cloud Run _ dlt Docs.md` (6.0K) was in
`docs/dlt/` (deleted with the `sync-skills-from-docs` change)
- The GCP Cloud Run docs: <https://cloud.google.com/run/docs>
- The dltHub BigQuery destination: <https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery>
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |