Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/cognee

cognee

Verified

references/infrastructure/INFRASTRUCTURE.md

View on GitHub
# Infrastructure — Supporting Stacks for the Cognition Pipeline

How Lakehouse, LakeFS, Dozzle, and Beszel support the documentation cognition pipeline with storage, versioning, logging, and monitoring.

## Lakehouse — Unified Data Storage

**Stack**: `infrastructure/stacks/lakehouse/`  
**Ports**: 3900-3904 (Garage S3), 8181 (Lakekeeper), 8182 (Lance Namespace)

### Role in the Cognition Pipeline

The lakehouse provides the storage backend for all cognition data:

| Data Type | Storage | Format |
|:--|:--|:--|
| Raw `.md` files | Garage S3 `docs/` bucket | Markdown text |
| Cognee vectors | LanceDB → Lance Namespace → Iceberg | Lance columnar |
| Knowledge graph | Neo4j (graphiti stack) | Property graph |
| Trace data | Langfuse (Postgres + ClickHouse) | Relational + columnar |
| Cognition metrics | Prometheus (monitoring stack) | Time-series |

### Why Lakehouse Matters

- **Single namespace**: All cognition data queryable through one Iceberg catalog — DuckDB can query both structured curriculum data AND Cognee's vector indexes through the Lance Namespace bridge
- **Time travel**: Iceberg snapshots let you query "what was in Cognee's knowledge graph after last month's cognify run?"
- **ACID transactions**: Concurrent writes from Dagster (cognify jobs) and reads from the web app (GraphRAG queries) without conflicts

### Deployment

```bash
cd infrastructure/stacks/lakehouse
docker compose up -d
# Services: garage, lakekeeper, lance-namespace, postgres
```

## Dozzle — Container Log Monitoring

**Stack**: `infrastructure/stacks/dozzle/`  
**Port**: varies (8080 in stack, Pangolin-routed)

### Role in the Cognition Pipeline

Dozzle provides real-time log visibility for all cognition containers:

| Container | What to Monitor |
|:--|:--|
| `cognee` | Ingestion progress, cognify status, LLM API errors |
| `graphiti-neo4j-1` | Graph database health, memory pressure |
| `falkordb` | Redis memory, connection errors |
| `langfuse-*` | Trace ingestion, ClickHouse write pressure |

### Key Log Patterns to Watch

```
# Cognee ingestion progress
[cognee] Ingesting 540 files → dataset 'docs-ml'
[cognee] [####----] 270/540 (50%)

# Cognify status
[cognee] Entity extraction started: 2,242 documents
[cognee] Relationship inference: 8,476 edges created
[cognee] Cognify complete: 12.3s, 847,000 tokens, $4.23 cost

# LLM API errors
[cognee] LLMAPIKeyNotSetError — check DEEPSEEK_API_KEY
[cognee] RateLimitError — reduce concurrent cognify batches
```

## Beszel — Server Metrics Dashboard

**Stack**: `infrastructure/stacks/beszel/`  
**Port**: 8090

### Role in the Cognition Pipeline

Beszel monitors resource utilization across all 3 physical hosts during cognition operations:

| Host | Monitored Resources | Why It Matters |
|:--|:--|:--|
| `cax41-hetzner` (32 GB) | Memory, CPU | Neo4j + FalkorDB + Langfuse run here; KG build is memory-intensive |
| `arm1-oci` (24 GB) | CPU, Disk | Lakehouse + LakeFS + Dozzle run here; S3 I/O during cognify |
| `bunchloch` (48 GB unified) | Memory, GPU | Cognee + Graphiti run here; LLM processing uses unified memory |

### Alerts to Configure

```
ALERT: bunchloch_memory > 40GB
  → Neo4j + Cognee + LLM model competing for unified memory
  → Action: Reduce concurrent cognify batches

ALERT: hetzner_disk > 80%
  → LanceDB vector indexes growing
  → Action: Prune old indexes, increase disk

ALERT: arm1_cpu > 90% sustained 5min
  → Lakehouse S3 operations saturating CPU
  → Action: Throttle concurrent data writes
```

## Combined Health Check

```bash
#!/bin/bash
# Check all cognition pipeline components

echo "=== Cognition Pipeline Health ==="

# Cognee
curl -s -o /dev/null -w "Cognee:    %{http_code}\n" http://localhost:8100/docs

# Neo4j (via Graphiti stack)
curl -s -o /dev/null -w "Neo4j:     %{http_code}\n" http://localhost:7474

# FalkorDB
curl -s -o /dev/null -w "FalkorDB:  %{http_code}\n" http://localhost:6379

# Lakehouse (Garage S3)
curl -s -o /dev/null -w "Garage:    %{http_code}\n" http://localhost:3900

# LakeFS
curl -s -o /dev/null -w "LakeFS:    %{http_code}\n" http://localhost:8000/health

# Dozzle
curl -s -o /dev/null -w "Dozzle:    %{http_code}\n" http://localhost:7070

# Beszel
curl -s -o /dev/null -w "Beszel:    %{http_code}\n" http://localhost:8090

# CCC index
ls -lh .cocoindex_code/target_sqlite.db | awk '{print "CCC:       " $5 " index"}'

echo "================================"
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall