fcakyon/claude-codex-settings / plugins/mongodb-skills/skills/mongodb-schema-design
mongodb-schema-design
VerifiedMongoDB schema design patterns and anti-patterns. Use when designing data models, reviewing schemas, migrating from SQL, or troubleshooting performance issues caused by schema problems. Triggers on "design schema", "embed vs reference", "MongoDB data model", "schema review", "unbounded arrays", "one-to-many", "tree structure", "16MB limit", "schema validation", "JSON Schema", "time series", "schema migration", "polymorphic", "TTL", "data lifecycle", "archive", "index explosion", "unnecessary indexes", "approximation pattern", "document versioning".
references/pattern-outlier.md
View on GitHub---
title: Use Outlier Pattern for Exceptional Documents
impact: MEDIUM
impactDescription: "Isolates unusually large documents so hot-path queries stay optimized for typical cases"
tags: schema, patterns, outlier, arrays, performance, edge-cases
---
## Use Outlier Pattern for Exceptional Documents
**Isolate atypical documents with large arrays to prevent them from degrading performance for typical queries.** When a small subset of documents is much larger than the rest, those outliers can dominate memory, index, and query costs. Split overflow data into a separate collection and flag the document.
**Problem scenario:**
A typical book might have 50 customers in an embedded array, while a bestseller like Harry Potter accumulates 50,000 (~2.5MB). Queries return the full document, so the outlier dominates memory and network cost. A multikey index on that array produces 50,000 entries for a single document.
**Correct (outlier pattern):**
Typical documents keep their full embedded array and set `hasExtras: false`. Outlier documents cap the embedded array at a threshold (e.g. 50), set `hasExtras: true`, store a denormalized `customerCount`, and overflow remaining items into a separate collection in batched documents (e.g. `{ bookId, customers: [...], batch: 1, count: 950 }`). Application code checks the `hasExtras` flag to decide whether to load overflow batches.
**Implementation with threshold (example; tune per workload):**
```javascript
const CUSTOMER_THRESHOLD = 50
async function addCustomer(bookId, customerId) {
// Try the normal case first: atomically add to the embedded array only if
// the current customerCount is below the threshold (treat missing/null as 0).
const result = await db.books.updateOne(
{
_id: bookId,
$or: [
{ customerCount: { $lt: CUSTOMER_THRESHOLD } },
{ customerCount: { $exists: false } },
{ customerCount: null }
]
},
{
$push: { customers: customerId },
$inc: { customerCount: 1 }
}
)
if (result.matchedCount > 0) {
// Normal case succeeded - customer added to embedded array
return
}
// Outlier case - add to overflow collection
const lastBatchDoc = await db.book_customers_extra
.find({ bookId: bookId })
.sort({ batch: -1 })
.limit(1)
.next()
const nextBatch = lastBatchDoc ? lastBatchDoc.batch + 1 : 1
const targetBatch =
lastBatchDoc && lastBatchDoc.count < 1000
? lastBatchDoc.batch
: nextBatch
// First, try to append to the intended batch, enforcing the 1000-item cap under concurrency.
const overflowFilter = { bookId: bookId, batch: targetBatch }
if (targetBatch !== nextBatch) {
// Only enforce the count cap when targeting an existing batch.
overflowFilter.count = { $lt: 1000 }
}
const overflowResult = await db.book_customers_extra.updateOne(
overflowFilter, // Write to the intended batch, respecting the count cap when reusing a batch
{
$push: { customers: customerId },
$inc: { count: 1 },
$setOnInsert: { bookId: bookId, batch: targetBatch }
},
{ upsert: targetBatch === nextBatch }
)
// If we failed to match when trying to reuse the previous batch (it filled concurrently),
// fall back to writing into the next batch.
if (overflowResult.matchedCount === 0 && targetBatch !== nextBatch) {
await db.book_customers_extra.updateOne(
{ bookId: bookId, batch: nextBatch },
{
$push: { customers: customerId },
$inc: { count: 1 },
$setOnInsert: { bookId: bookId, batch: nextBatch }
},
{ upsert: true }
)
}
await db.books.updateOne(
{ _id: bookId },
{
$set: { hasExtras: true },
$inc: { customerCount: 1 }
}
)
}
```
**Index strategy:**
```javascript
// Index on main collection - only 50 entries per outlier doc
db.books.createIndex({ "customers": 1 })
// Index on overflow collection
db.book_customers_extra.createIndex({ bookId: 1 })
db.book_customers_extra.createIndex({ customers: 1 })
```
**When to use outlier pattern:**
| Scenario | What to measure | Example |
|----------|-----------------|---------|
| Book customers | Array-size distribution and long tail | Bestsellers vs. typical books |
| Social followers | Growth rate and read-path impact | Celebrities vs. regular users |
| Product reviews | Index fan-out and read locality | Viral products vs. typical |
| Event attendees | Outlier frequency vs. implementation complexity | Major events vs. small meetups |
**When NOT to use this pattern:**
- **Uniform distribution**: If all documents have similar array sizes, no outliers to isolate.
- **Always need full data**: If you always display all 50,000 customers, pattern doesn't help.
- **Write-heavy outliers**: Complex update logic may not be worth the read optimization.
- **Small outliers**: If outliers are 200 vs typical 50, just use larger threshold.
## Verify with
```javascript
// Find outlier documents
db.books.aggregate([
{ $project: {
title: 1,
customerCount: { $size: { $ifNull: ["$customers", []] } }
}},
{ $sort: { customerCount: -1 } },
{ $limit: 20 }
])
// Calculate distribution
db.books.aggregate([
{ $project: { count: { $size: { $ifNull: ["$customers", []] } } } },
{ $bucket: {
groupBy: "$count",
boundaries: [0, 50, 100, 500, 1000, 10000, 100000],
default: "100000+",
output: { count: { $sum: 1 } }
}}
])
// Look for a long-tail distribution where a small subset is far above median/p95
// Check index sizes
db.books.stats().indexSizes
// Large multikey index suggests outliers are bloating it
```
Reference: [Outlier Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/group-data/outlier-pattern/)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |