fcakyon/claude-codex-settings / plugins/mongodb-skills/skills/mongodb-atlas-stream-processing
mongodb-atlas-stream-processing
VerifiedManages MongoDB Atlas Stream Processing (ASP) workflows. Handles workspace provisioning, data source/sink connections, processor lifecycle operations, debugging diagnostics, and tier sizing. Supports Kafka, Atlas clusters, S3, HTTPS, and Lambda integrations for streaming data workloads and event processing. NOT for general MongoDB queries or Atlas cluster management. Requires MongoDB MCP Server with Atlas API credentials.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.4 KB |
| references/connection-configs.md | markdown | 12.8 KB |
| references/development-workflow.md | markdown | 15.0 KB |
| references/mcp-troubleshooting.md | markdown | 1.8 KB |
| references/output-diagnostics.md | markdown | 6.6 KB |
| references/pipeline-patterns.md | markdown | 16.1 KB |
| references/sizing-and-parallelism.md | markdown | 6.1 KB |
references/sizing-and-parallelism.md
View on GitHub# Sizing & Parallelism Reference
## Tier Hardware Specs
| Tier | vCPU | RAM | Bandwidth | Max Parallelism | Kafka Partitions | Use case |
|------|------|-----|-----------|-----------------|------------------|----------|
| SP2 | 0.25 | 512MB | 50 Mbps | 1 | 32 | Minimal filtering, testing |
| SP5 | 0.5 | 1GB | 125 Mbps | 2 | 64 | Simple filtering and routing |
| SP10 | 1 | 2GB | 200 Mbps | 8 | Unlimited | Moderate workloads, joins, grouping |
| SP30 | 2 | 8GB | 750 Mbps | 16 | Unlimited | Windows, JavaScript UDFs, production |
| SP50 | 8 | 32GB | 2500 Mbps | 64 | Unlimited | High throughput, large window state |
**Memory rule:** 20% is reserved for overhead. User state (window accumulation, sort buffers) must stay below 80% of tier RAM. Exceeding this causes OOM failure.
## How Parallelism Works
Every stage in a pipeline runs with default `parallelism: 1`. This base level is included in your tier at no additional cost.
When you need higher throughput for specific stages, increase their parallelism beyond 1. **Only values > 1 count toward your tier's maximum.**
Stages that commonly benefit from parallelism:
- `$merge` — concurrent writes to Atlas
- `$lookup` — concurrent reads for enrichment
- `$https` — concurrent API calls
## Parallelism Calculation
**Formula:** `Total Parallelism = sum of (parallelism - 1) for all stages where parallelism > 1`
### Tier Selection Algorithm
```
If Total Parallelism = 0: → SP2 (max 1)
If Total Parallelism = 1: → SP5 (max 2)
If Total Parallelism ≤ 8: → SP10 (max 8)
If Total Parallelism ≤ 16: → SP30 (max 16)
If Total Parallelism ≤ 64: → SP50 (max 64)
```
### Worked Examples
**Simple pipeline (all parallelism = 1):**
```
$source: parallelism = 1 (does not count)
$match: parallelism = 1 (does not count)
$merge: parallelism = 1 (does not count)
Total = 0 → SP2
```
**Medium pipeline:**
```
$source: parallelism = 1 (does not count)
$match: parallelism = 1 (does not count)
$lookup: parallelism = 4 (counts as 3)
$merge: parallelism = 4 (counts as 3)
Total = 3 + 3 = 6 → SP10 (max 8)
```
**Complex pipeline:**
```
$source: parallelism = 1 (does not count)
$https: parallelism = 6 (counts as 5)
$merge: parallelism = 8 (counts as 7)
Total = 5 + 7 = 12 → SP30 (max 16)
```
### API Error for Parallelism Exceeded
If you specify a tier too small for the pipeline's parallelism, the API returns:
```
"Operator parallelism requested exceeds limit for this tier.
(Requested: X, Limit: Y). Minimum tier for this workload: SPxx or larger."
```
Solution: Use `atlas-streams-manage` → `stop-processor`, then `start-processor` with a higher `tier` value.
## Complexity-Based Tier Selection
When parallelism is all default (1), choose tier based on pipeline complexity:
| Pipeline feature | Complexity weight | Minimum tier |
|-----------------|-------------------|--------------|
| Simple `$match` + `$project` only | Low | SP2-SP5 |
| `$addFields` with expressions | Low-Medium | SP5-SP10 |
| `$lookup` or `$https` enrichment | Medium | SP10 |
| `$group` aggregation | Medium | SP10 |
| `$tumblingWindow` or `$hoppingWindow` | Medium-High | SP10-SP30 |
| `$sessionWindow` | High | SP30 |
| `$function` (JavaScript UDFs) | High | SP30+ |
| Large window state (many unique keys) | Very High | SP30-SP50 |
| Multiple windows or chained enrichment | Very High | SP50 |
### Complexity Scoring Heuristic
For automated tier recommendation, score the pipeline:
| Feature | Points |
|---------|--------|
| `$function` (JavaScript) | +40 |
| Window operations (`$tumblingWindow`, `$hoppingWindow`, `$sessionWindow`) | +30 |
| `$lookup` or `$https` enrichment | +20 |
| `$group` aggregation | +15 |
| Kafka source integration | +15 |
| `$sort` operations | +10 |
| Pipeline has 5+ stages | +5 |
| Pipeline has 8+ stages | +10 |
| Pipeline has 12+ stages | +20 |
**Score → Tier mapping:**
- 0-10: SP2
- 11-20: SP5
- 21-40: SP10
- 41-60: SP30
- 61+: SP50
**Always take the higher of complexity-driven vs parallelism-driven tier recommendations.**
## Billing
Charges are **per-hour, calculated per-second**, only while the processor is running.
- `start-processor` begins billing
- `stop-processor` stops billing
- Stopped processors retain state for 45 days at no charge
**What's included in the tier price:**
- Compute (vCPU and RAM)
- State storage
- Base parallelism (parallelism = 1 for all stages)
**Additional costs (separate from tier):**
- Data transfer egress (varies by cloud provider and transfer type: intra-region, inter-region, internet)
- VPC Peering (AWS and GCP)
- Private Link connectivity
For current pricing: https://www.mongodb.com/docs/atlas/billing/stream-processing-costs/
## Sizing Workflow with MCP Tools
### Phase 1: Pre-deployment estimate
1. Score the pipeline using the complexity heuristic above
2. Calculate parallelism needs using the formula
3. Take the higher recommendation
4. Start with that tier (or one tier lower for cost savings during testing)
### Phase 2: Validation
1. Deploy the processor: `atlas-streams-build` → `resource: "processor"` with `autoStart: true`
2. Let it run for a representative period
3. Check stats: `atlas-streams-discover` → `diagnose-processor`
4. Review `memoryUsageBytes`:
- Below 50% of tier RAM → over-provisioned, consider downsizing
- 50-70% → good fit
- 70-80% → at limit, monitor closely
- Above 80% → under-provisioned, upgrade before it OOMs
### Phase 3: Optimization
1. Stop processor: `atlas-streams-manage` → `stop-processor`
2. Restart with adjusted tier: `atlas-streams-manage` → `start-processor` with `tier` override
3. Monitor for another period
4. Repeat until right-sized
### Cost Optimization: Time-of-Day Strategy
For workloads with predictable traffic patterns, adjust tiers by time of day:
| Period | Tier | Rationale |
|--------|------|-----------|
| Peak hours (business hours) | SP30-SP50 | Handle full volume |
| Off-peak hours | SP10-SP30 | Reduced volume |
| Maintenance windows | SP2-SP10 | Minimal processing |
To change tiers: `stop-processor` → `start-processor` with new `tier` value. Note: `resumeFromCheckpoint: true` (default) preserves state across tier changes.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |