artubss/SKILLS-CLAUDE-CODE / favorites/tpl-ai-ml-vector-search-pinecone
tpl-ai-ml-vector-search-pinecone
VerifiedTemplate do pack (ai-ml/07-vector-search-pinecone.md). Orienta o agente em integracao de IA/ML, LLM e pipelines de dados alinhado a esse contexto.
PROJECT: Vector Search with Pinecone (Node.js/Python + OpenAI Embeddings)
Skill gerado a partir do pack templates-claude-code. Arquivo de origem: ai-ml/07-vector-search-pinecone.md. Use como baseline e adapte ao projeto antes de mudancas grandes.
Conteudo do template
STACK
- Vector DB: Pinecone SDK v3 (pinecone-client)
- Language: Node.js + TypeScript (primary) / Python (ingestion scripts)
- Embeddings: OpenAI
text-embedding-3-small(1536 dims) - Hybrid Search: Pinecone sparse+dense (combined score)
- Metadata Filtering: Pinecone metadata filters (JSON)
- Cost Optimization: Batch upserts + namespace strategies
PROJECT STRUCTURE
pinecone-app/
├── src/
│ ├── pinecone/
│ │ ├── client.ts # Pinecone singleton
│ │ ├── upsert.ts # Batch upsert logic
│ │ ├── query.ts # Query + filter patterns
│ │ └── namespaces.ts # Namespace strategy
│ ├── embeddings/
│ │ └── openai.ts # Embedding generation + caching
│ └── types/
│ └── vector.ts # TypeScript interfaces
├── scripts/
│ ├── ingest.py # Python bulk ingest
│ ├── backup.py # Index backup to S3
│ └── cleanup.py # Delete old vectors
└── tests/
└── pinecone.test.ts
ARCHITECTURE RULES
- Namespace per data category — not per user (too many namespaces); use metadata for user filtering.
- Batch upserts: 100 vectors/request — Pinecone free tier: 100/req; paid: up to 1000. Always batch.
- Metadata is small, < 40KB/vector — don't store full documents in metadata; store IDs + retrieval keys.
- Dimensions match embedding model —
text-embedding-3-small= 1536;text-embedding-3-large= 3072. Can't mix. - Cosine similarity for text — always
metric: cosinefor text embeddings;euclideanfor image embeddings. - Filter before score threshold — apply metadata filters in query, not post-query; cheaper + more accurate.
- Cache embeddings for repeated queries — same query text = same embedding; cache in Redis with SHA256 key.
- Hybrid search for keyword+semantic mix — use Pinecone sparse+dense for product search, FAQ matching.
INDEX CONFIGURATION
// src/pinecone/client.ts
import { Pinecone } from '@pinecone-database/pinecone'
if (!process.env.PINECONE_API_KEY) throw new Error('PINECONE_API_KEY required')
export const pinecone = new Pinecone({
apiKey: process.env.PINECONE_API_KEY,
})
// Index configuration (create once, use always)
export const INDEX_CONFIG = {
name: 'my-app-vectors',
dimension: 1536, // text-embedding-3-small dimensions
metric: 'cosine' as const, // Always cosine for text
spec: {
serverless: {
cloud: 'aws',
region: 'us-east-1',
},
},
}
export async function ensureIndex(): Promise<void> {
const { indexes } = await pinecone.listIndexes()
const exists = indexes?.some(i => i.name === INDEX_CONFIG.name)
if (!exists) {
console.log(`Creating index: ${INDEX_CONFIG.name}`)
await pinecone.createIndex({
...INDEX_CONFIG,
waitUntilReady: true,
})
}
}
export const index = pinecone.index(INDEX_CONFIG.name)
NAMESPACE STRATEGY
// src/pinecone/namespaces.ts
/**
* Namespace per logical data type — NOT per user
*
* ✅ Good namespace design:
* "docs-v1" → Product documentation (version tagged)
* "kb-articles" → Knowledge base articles
* "products" → Product catalog embeddings
* "support-tickets"→ Historical support tickets
*
* ❌ Bad namespace design:
* "user-123" → Per-user (creates thousands of namespaces)
* "2024-01" → Per month (stale data hard to query)
*
* Use metadata for per-user/per-org filtering:
* metadata: { org_id: "org-456", visibility: "private" }
*/
export const NAMESPACES = {
DOCS: 'docs-v1',
KB: 'kb-articles',
PRODUCTS: 'products',
SUPPORT: 'support-tickets',
} as const
export type Namespace = typeof NAMESPACES[keyof typeof NAMESPACES]
EMBEDDING GENERATION + CACHING
// src/embeddings/openai.ts
import OpenAI from 'openai'
import { createHash } from 'crypto'
import { Redis } from 'ioredis'
const openaiClient = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
const redis = new Redis(process.env.REDIS_URL || 'redis://localhost:6379')
const EMBED_CACHE_TTL = 86400 // 24 hours — embeddings don't change
export async function embed(text: string): Promise<number[]> {
const key = `embed:${createHash('sha256').update(text).digest('hex').slice(0, 16)}`
const cached = await redis.get(key)
if (cached) return JSON.parse(cached)
const response = await openaiClient.embeddings.create({
model: 'text-embedding-3-small',
input: text,
encoding_format: 'float',
})
const embedding = response.data[0].embedding
await redis.setex(key, EMBED_CACHE_TTL, JSON.stringify(embedding))
return embedding
}
export async function embedBatch(texts: string[]): Promise<number[][]> {
// OpenAI supports up to 2048 inputs per batch
const BATCH = 100
const results: number[][] = []
for (let i = 0; i < texts.length; i += BATCH) {
const batch = texts.slice(i, i + BATCH)
const response = await openaiClient.embeddings.create({
model: 'text-embedding-3-small',
input: batch,
encoding_format: 'float',
})
results.push(...response.data.map(d => d.embedding))
}
return results
}
BATCH UPSERT
// src/pinecone/upsert.ts
import { index, NAMESPACES, type Namespace } from './client'
import { embedBatch } from '../embeddings/openai'
import type { PineconeRecord, RecordMetadata } from '@pinecone-database/pinecone'
interface UpsertDocument {
id: string
text: string
metadata: RecordMetadata
}
export async function upsertDocuments(
documents: UpsertDocument[],
namespace: Namespace = NAMESPACES.DOCS,
batchSize: number = 100,
): Promise<number> {
const ns = index.namespace(namespace)
let total = 0
for (let i = 0; i < documents.length; i += batchSize) {
const batch = documents.slice(i, i + batchSize)
// Embed the batch
const embeddings = await embedBatch(batch.map(d => d.text))
// Build Pinecone records
const records: PineconeRecord[] = batch.map((doc, idx) => ({
id: doc.id,
values: embeddings[idx],
metadata: {
...doc.metadata,
text: doc.text.slice(0, 500), // Store truncated text for retrieval
indexed_at: new Date().toISOString(),
},
}))
await ns.upsert(records)
total += batch.length
console.log(`Upserted ${total}/${documents.length} vectors`)
}
return total
}
QUERY + METADATA FILTERING
// src/pinecone/query.ts
import { index, type Namespace } from './client'
import { embed } from '../embeddings/openai'
interface QueryOptions {
topK?: number
namespace?: Namespace
filter?: Record<string, unknown>
includeMetadata?: boolean
scoreThreshold?: number
}
export async function semanticSearch(
query: string,
options: QueryOptions = {},
) {
const {
topK = 10,
namespace = 'docs-v1',
filter,
includeMetadata = true,
scoreThreshold = 0.72, // Reject irrelevant results
} = options
const queryEmbedding = await embed(query)
const ns = index.namespace(namespace)
const response = await ns.query({
vector: queryEmbedding,
topK,
filter, // Applied server-side: faster + cheaper
includeMetadata,
includeValues: false, // Don't return raw vectors: reduces payload
})
// Apply score threshold
return response.matches
.filter(m => (m.score ?? 0) >= scoreThreshold)
.map(m => ({
id: m.id,
score: m.score,
metadata: m.metadata,
}))
}
// Filtered search example:
// const results = await semanticSearch("deployment guide", {
// filter: {
// org_id: { $eq: "org-456" },
// doc_type: { $in: ["guide", "tutorial"] },
// version: { $gte: 2 },
// }
// })
HYBRID SEARCH (SPARSE + DENSE)
// Pinecone sparse-dense hybrid (requires dotproduct metric index)
export async function hybridSearch(
query: string,
sparseValues: { indices: number[]; values: number[] }, // From BM25 encoder
options: QueryOptions = {},
) {
const denseVector = await embed(query)
const ns = index.namespace(options.namespace || 'products')
// alpha: 0 = pure sparse, 1 = pure dense, 0.5 = balanced
const alpha = 0.5
const scaledDense = denseVector.map(v => v * alpha)
const scaledSparse = {
indices: sparseValues.indices,
values: sparseValues.values.map(v => v * (1 - alpha)),
}
return ns.query({
vector: scaledDense,
sparseVector: scaledSparse,
topK: options.topK || 10,
includeMetadata: true,
})
}
INDEX BACKUP TO S3
# scripts/backup.py
import boto3
import json
from pinecone import Pinecone
from datetime import datetime
import os
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
s3 = boto3.client("s3")
BUCKET = "my-app-pinecone-backups"
INDEX_NAME = "my-app-vectors"
def backup_index():
index = pc.Index(INDEX_NAME)
stats = index.describe_index_stats()
namespaces = list(stats.namespaces.keys())
backup_data = {
"timestamp": datetime.utcnow().isoformat(),
"index_name": INDEX_NAME,
"stats": stats.to_dict(),
"namespaces": namespaces,
}
key = f"backups/{INDEX_NAME}/{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}/metadata.json"
s3.put_object(
Bucket=BUCKET,
Key=key,
Body=json.dumps(backup_data),
)
print(f"Backup metadata saved to s3://{BUCKET}/{key}")
# Note: Pinecone doesn't support full vector export in free tier
# For paid: export to parquet using Pinecone serverless export feature
if __name__ == "__main__":
backup_index()
COST OPTIMIZATION RULES
Index Type │ Cost │ Use When
─────────────┼────────────┼──────────────────────────────────
Serverless │ Per usage │ Variable traffic, dev/staging
Pods (p1) │ Fixed/hour │ Consistent high traffic, SLA needed
Pods (s1) │ $ │ Storage-heavy, few queries
Optimization strategies:
1. Reduce dimensions: text-embedding-3-small (1536) vs large (3072) — 2x cheaper queries
2. Delete stale vectors: run cleanup_old_vectors() weekly
3. Use namespaces to isolate high-query from low-query data
4. Cache query embeddings in Redis (same query = no re-embed)
5. Sparse+dense on paid plan only — avoid if on free tier
6. topK=10 max for interactive; topK=50 for batch — each match has cost
Cost formula: Reads × $0.000000125 per unit (1000 dims × 1 vector = 1 unit)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |