Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/cognee

cognee

Verified

references/cognee.md

View on GitHub
---
name: cognee
description: Creating and developing AI memory systems with Cognee. Use when working with knowledge graphs, semantic search, or building persistent AI agent memory. (project, ai-memory)
category: AI Memory
tags: [cognee, knowledge-graph, rag, vector-search, graph-database, ai-memory]
---

# Cognee AI Memory Expert

You are an expert in using Cognee, an open-source AI memory platform that transforms raw data into persistent, dynamic memory for AI agents.

## Core Capabilities

When users request help with Cognee, you should assist with:

### 1. Setup and Configuration

```python
import cognee
import os

# LLM Configuration
os.environ["LLM_API_KEY"] = "your-api-key"
await cognee.config.set_llm_provider("openai")
await cognee.config.set_llm_model("gpt-4o-mini")

# Graph Database Configuration
await cognee.config.set_graph_database_provider("neo4j")
await cognee.config.set_graph_database_url("bolt://localhost:7687")
await cognee.config.set_graph_database_username("neo4j")
await cognee.config.set_graph_database_password("password")

# Vector Database Configuration
await cognee.config.set_vector_database_provider("lancedb")
await cognee.config.set_vector_database_url("./lancedb_data")

# Embedding Configuration
await cognee.config.set_embedding_provider("openai")
await cognee.config.set_embedding_model("text-embedding-3-large")
```

### 2. ECL Pipeline (Extract-Cognify-Load)

The fundamental pattern for building AI memory:

```python
# Extract: Add data to cognee
await cognee.add(content, dataset_name="my_dataset")

# Cognify: Transform into knowledge graph
await cognee.cognify()

# Load: Automatically stored in configured databases
# Search: Query the knowledge graph
results = await cognee.search("Your question", query_type=SearchType.GRAPH_COMPLETION)
```

### 3. Data Ingestion Patterns

**Single Document:**
```python
await cognee.add("Your document text", dataset_name="docs")
```

**Multiple Documents:**
```python
documents = ["doc1", "doc2", "doc3"]
for doc in documents:
    await cognee.add(doc, dataset_name="knowledge_base")
await cognee.cognify()
```

**With Metadata:**
```python
# Primary content
await cognee.add(content, dataset_name="posts")

# Associated metadata
await cognee.add(metadata, dataset_name="post_metadata")

await cognee.cognify()
```

**File Upload:**
```python
with open("document.pdf", "rb") as f:
    await cognee.add(f, dataset_name="pdfs")
```

### 4. Search Strategies

Cognee provides multiple search types for different use cases:

```python
from cognee.api.v1.search import SearchType

# Semantic vector search (fast, similarity-based)
results = await cognee.search(
    query_text="find similar concepts",
    query_type=SearchType.CHUNKS
)

# Graph-based insights (relationship-aware)
results = await cognee.search(
    query_text="how are these concepts related",
    query_type=SearchType.INSIGHTS
)

# Hybrid search with LLM reasoning (most powerful)
results = await cognee.search(
    query_text="complex multi-hop question",
    query_type=SearchType.GRAPH_COMPLETION,
    top_k=5
)

# Document summaries (hierarchical)
results = await cognee.search(
    query_text="summarize the main themes",
    query_type=SearchType.SUMMARIES
)

# Code search (language-aware)
results = await cognee.search(
    query_text="authentication implementation",
    query_type=SearchType.CODE
)

# Direct graph queries (Cypher)
results = await cognee.search(
    query_text="MATCH (n:Entity) RETURN n",
    query_type=SearchType.CYPHER
)

# Automatic search type selection
results = await cognee.search(
    query_text="your question",
    query_type=SearchType.FEELING_LUCKY
)
```

### 5. Dataset Management

**Scoped Queries:**
```python
# Add to specific datasets
await cognee.add(data1, dataset_name='dataset_a')
await cognee.add(data2, dataset_name='dataset_b')

# Cognify all datasets
await cognee.cognify()

# Search specific dataset
results = await cognee.search(
    query_text="query",
    node_name="dataset_a",
    top_k=5
)
```

**Data Cleanup:**
```python
# Clear all data
await cognee.prune.prune_data()

# Full system reset
await cognee.prune.prune_system(metadata=True)

# Delete specific data
await cognee.delete(data_id)
```

### 6. Visualization

```python
# Generate static visualization
await cognee.visualize_graph('/path/to/graph_visualization.html')

# Start interactive visualization server
await cognee.start_visualization_server()

# Network visualization
await cognee.cognee_network_visualization()
```

### 7. Integration Patterns

**DLT Integration:**
```python
import dlt
import cognee

@dlt.resource(write_disposition="merge", primary_key="id")
def data_source():
    yield data

# Load with DLT
pipeline = dlt.pipeline(
    pipeline_name="cognee_pipeline",
    destination="duckdb"
)
pipeline.run(data_source())

# Process with Cognee
await cognee.add(data, dataset_name="dlt_dataset")
await cognee.cognify()
```

**LangGraph Integration:**
```python
from langgraph.graph import StateGraph
import cognee

# Use cognee as memory backend
async def retrieve_context(state):
    results = await cognee.search(state["query"])
    return {"context": results}

workflow = StateGraph(...)
workflow.add_node("memory", retrieve_context)
```

**MCP (Model Context Protocol):**
```python
from cognee import MCP

# Expose cognee as MCP server
cognee_server = MCP()
cognee_server.start()

# Available functions:
# - cognify: Transform text into knowledge graphs
# - save_interaction: Capture conversations
# - search: Multi-mode semantic search
# - list_data: Display stored datasets
# - delete: Remove specific data
# - prune: Full memory reset
```

### 8. Incremental Updates

```python
# Initial load
await cognee.add(initial_data, dataset_name="docs")
await cognee.cognify()

# Later: add new data incrementally
# Cognee intelligently updates the existing graph
await cognee.add(new_documents, dataset_name="docs")
await cognee.cognify()  # Only processes new/changed content
```

### 9. Advanced Patterns

**Batch Processing:**
```python
from cognee.modules.pipelines import Task

pipeline = [
    Task(name="extract", batch_size=100),
    Task(name="chunk", batch_size=50),
    Task(name="embed", batch_size=20),
    Task(name="extract_entities"),
    Task(name="build_graph"),
]
```

**Custom Ontologies:**
Cognee supports custom business ontologies for domain-specific entity extraction and relationship mapping through pipeline configuration.

**Distributed Processing:**
For large-scale deployments, use distributed Cognee for parallel dataset processing across multiple workers.

## Architecture Knowledge

### Three Knowledge Layers
1. **Raw Information Nodes**: Original document content
2. **Extracted Entities**: Concepts and objects identified
3. **Relationship Mappings**: Connections between entities

### Storage Systems

**Vector Stores** (semantic search):
- LanceDB (default, local)
- Qdrant Cloud
- PGVector (Postgres)
- Weaviate
- FalkorDB
- Redis

**Graph Databases** (relationships):
- KuzuDB (default, embedded)
- Neo4j
- Neptune (AWS)
- Memgraph
- NetworkX (in-memory)

**Relational** (metadata):
- PostgreSQL
- SQLite

### Knowledge Graph Ontology

**Node Types:**
- Entity Nodes: Concepts extracted from text
- Document Nodes: Source documents
- Chunk Nodes: Text segments with embeddings
- Metadata Nodes: Supplementary information

**Relationship Types:**
- `RELATIONSHIP`: Subject-Object connections
- `MENTION`: Document references entity
- `related_to`: General conceptual relationships
- `contains`: Hierarchical containment
- `hasClause`: Domain-specific

## Common Use Cases

1. **Conversational AI Memory**: Persistent context across chat sessions
2. **Document Q&A**: Transform documentation into queryable knowledge graphs
3. **Code Intelligence**: Semantic code search and understanding
4. **Research Analysis**: Knowledge discovery from large document sets
5. **Data Unification**: Consolidate scattered data silos
6. **Enterprise Search**: Precise, cited answers with sources
7. **Multi-Agent Memory**: Shared knowledge base for agent teams
8. **Vertical AI Agents**: Domain-specific copilots that learn

## Best Practices

1. **Use Datasets for Organization**: Group related content in named datasets
2. **Choose Right Search Type**: Use CHUNKS for speed, GRAPH_COMPLETION for depth
3. **Incremental Processing**: Add data incrementally rather than full reloads
4. **Async Operations**: All Cognee operations are async - use `await`
5. **Configure Before Adding**: Set up LLM and databases before ingesting data
6. **Clean Up Old Data**: Use `prune` commands to manage memory usage
7. **Visualize Graphs**: Use visualization tools to understand knowledge structure
8. **Batch Large Datasets**: Configure batch sizes for large data processing
9. **Monitor Performance**: Use `top_k` to limit result sets and improve speed
10. **Leverage MCP**: Use MCP integration for standardized AI agent access

## Troubleshooting

**Graph database connection fails:**
```python
# Verify configuration
await cognee.config.set_graph_database_url("bolt://localhost:7687")
await cognee.config.set_graph_database_username("neo4j")
await cognee.config.set_graph_database_password("password")
```

**Rate limiting from LLM provider:**
```python
# Reduce batch size
Task(name="embed", batch_size=10)
```

**High memory usage:**
```python
# Clear old data
await cognee.prune.prune_data()

# Or selectively delete
await cognee.delete(data_id)
```

**Search returns no results:**
```python
# Verify data was cognified
await cognee.cognify()

# Try different search type
results = await cognee.search(
    query_text="your question",
    query_type=SearchType.FEELING_LUCKY
)
```

## Environment Variables

```bash
# LLM Configuration
export LLM_API_KEY="your-openai-key"
export LLM_PROVIDER="openai"
export LLM_MODEL="gpt-4o-mini"

# Graph Database
export GRAPH_DATABASE_PROVIDER="neo4j"
export GRAPH_DATABASE_URL="bolt://localhost:7687"
export GRAPH_DATABASE_USERNAME="neo4j"
export GRAPH_DATABASE_PASSWORD="password"

# Vector Database
export VECTOR_DATABASE_PROVIDER="lancedb"
export VECTOR_DATABASE_URL="./lancedb_data"

# Embedding
export EMBEDDING_PROVIDER="openai"
export EMBEDDING_MODEL="text-embedding-3-large"
```

## Quick Reference

```python
import cognee
from cognee.api.v1.search import SearchType

# Setup
await cognee.config.set_llm_provider("openai")
await cognee.config.set_llm_api_key("your-key")

# Add data
await cognee.add(content, dataset_name="docs")

# Build knowledge graph
await cognee.cognify()

# Search
results = await cognee.search(
    query_text="your question",
    query_type=SearchType.GRAPH_COMPLETION
)

# Visualize
await cognee.visualize_graph('/path/to/graph.html')

# Clean up
await cognee.prune.prune_data()
```

## Guidelines for Helping Users

1. **Start Simple**: Begin with basic ECL pattern (add → cognify → search)
2. **Understand Use Case**: Ask about data types, scale, and search requirements
3. **Configure Appropriately**: Help choose right databases and LLM providers
4. **Show Examples**: Provide working code examples from the patterns above
5. **Explain Search Types**: Help users choose the right search strategy
6. **Debug Systematically**: Check configuration, data ingestion, cognify step
7. **Optimize Performance**: Suggest batch sizes, incremental updates, pruning
8. **Integrate Wisely**: Show how to combine with their existing stack (DLT, Dagster, LangGraph)
9. **Visualize Results**: Encourage visualization to understand graph structure
10. **Reference Documentation**: Point to https://docs.cognee.ai for details

## Resources

- **Documentation**: https://docs.cognee.ai
- **GitHub**: https://github.com/topoteretes/cognee
- **Website**: https://www.cognee.ai
- **License**: Apache 2.0
- **Python Support**: 3.10-3.13

## Installation

```bash
# Basic installation
pip install cognee

# With optional dependencies
pip install cognee[neo4j,postgres]

# CLI
cognee-cli -ui  # Launch full UI
cognee-cli add "text"
cognee-cli cognify
cognee-cli search "query"
```

---

## When This Skill Should Be Used

Invoke this skill when the user:
- Asks about knowledge graphs or graph databases
- Wants to build RAG systems or semantic search
- Needs persistent AI agent memory
- Is working with Cognee specifically
- Wants to integrate vector and graph databases
- Needs to process documents into queryable knowledge
- Is building conversational AI with memory
- Wants to unify data silos into knowledge graphs
- Asks about LlamaIndex, LangGraph, or MCP integrations with memory

## What This Skill Provides

- Complete Cognee API reference and patterns
- ECL pipeline best practices
- Search strategy guidance
- Integration patterns (DLT, LangGraph, MCP)
- Configuration examples
- Troubleshooting steps
- Performance optimization tips
- Real-world use case examples
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall