Skip to content

teaceo-debug/DEEP6 / .claude/skills/deep6-edge-research

deep6-edge-research

Verified

knowledge.md

View on GitHub
# DEEP6 Edge Research — Master Knowledge Base

## 1. Project Context

**System**: DEEP6 — NQ futures order flow trading system
**Thesis**: Absorption and exhaustion are the highest-alpha reversal signals in order flow.
**Live path**: NinjaTrader 8 (NT8) with Rithmic L2 DOM data
**Research path**: Python, deep6v2 package, 1yr OHLCV CSV + Databento MBO

The backtesting pipeline runs on **two data layers**:

| Layer | Data | Coverage | Use |
|-------|------|----------|-----|
| OHLCV (available now) | `data/backtests/nq_1yr_1m.csv` | Jan 2025 → Apr 2026 | Synthesized footprint signals |
| MBO (when downloaded) | `data/databento/nq_mbo/raw_dbn/` | Target: 3 weeks | True order-lifecycle signals |

---

## 2. Data Assets

### Available now (no key needed)
```
data/backtests/nq_1yr_1m.csv          # 458k 1-min RTH bars, Jan 2025–Apr 2026
data/backtests/nq_3mo_1m.csv          # 96k bars, ~3 months subset
data/backtests/signal_events.csv      # ~1.5M signal fires (run signal_collect.py)
data/backtests/analysis/              # Attribution outputs from signal_analyze.py
```

### Databento MBO (requires API key in .env)
```
DATABENTO_API_KEY=db-...  → add to .env
# Download 3 weeks:
python scripts/databento/download_nq_mbo.py --start 2026-04-28 --end 2026-05-19
# Output: data/databento/nq_mbo/raw_dbn/NQ_c_0_mbo_2026-04-28_2026-05-19.dbn.zst
```

### Existing results
```
data/backtests/zones_1yr_variant_C_equity.png   # PF 4.48, WR 76.7%, Sharpe 8.68
data/backtests/tier3_followsignal_A_trades.csv  # Tier3 follow-signal trades
scripts/results_zones_1yr.txt                   # Full zones strategy report
```

---

## 3. Infrastructure Map

### Core pipeline scripts
```
scripts/signal_collect.py    # Step 1: collect all signal fires → signal_events.csv (~3 min)
scripts/signal_analyze.py    # Step 2: fast analysis → attribution tables + charts
scripts/signal_attribution.py # All-in-one (slower, for reference)
scripts/backtest_zones_1yr.py # Zones entry strategy (best known result: Variant C/D)
```

### Python packages
```
deep6v2/signals/registry.py        # DetectorRegistry — all 12 detector classes
deep6v2/signals/absorption.py      # ABS_01..04 — 4 absorption variants
deep6v2/signals/exhaustion.py      # EXH_01..06 — 6 exhaustion variants
deep6v2/signals/imbalance.py       # IMB_01..09 — 9 imbalance types
deep6v2/signals/delta.py           # DELT_01..11 — 11 delta signals
deep6v2/signals/auction.py         # AUCT_01..05 — 5 auction theory signals
deep6v2/signals/trap.py            # TRAP_01..05 — 5 trapped trader signals (weight=0 currently)
deep6v2/signals/vol_patterns.py    # VOLP_01..06 — 6 volume pattern signals
deep6v2/signals/engines/           # ENG_02..07 — DOM depth engines
deep6v2/backtest/replay_engine.py  # Full replay with trade simulation
deep6v2/backtest/ohlcv_synthesizer.py # OHLCV → synthetic FootprintBar
deep6v2/backtest/trade_simulator.py   # Trade entry/exit/P&L
deep6v2/scoring/scorer.py             # ConfluenceScorer — synthesizes all signals
```

### Current scoring weights (ScoringConfig — may need reoptimization)
```python
absorption_weight   = 20.0  # currently scored highest
exhaustion_weight   = 15.7
imbalance_weight    = 25.0  # highest weight
delta_weight        = 14.3
volume_profile_weight = 20.2
auction_weight      = 12.6
trapped_weight      = 0.0   # DISABLED — not enough evidence
poc_weight          = 0.0   # DISABLED
# Confluence multiplier: 1.25x when 5+ categories agree same direction
# Midday block: bars 60-210 per session (11:00-14:30 ET approx)
# TypeA threshold: 80, TypeB: 72, TypeC: 50
```

---

## 4. Signal Taxonomy

### 8 categories, 44 signals + engines

| Category | Signals | What they detect |
|----------|---------|-----------------|
| ABSORPTION | ABS_01-04 | Buyers/sellers absorbing aggression without giving ground |
| EXHAUSTION | EXH_01-06 | Momentum running out — thin prints, fading delta |
| IMBALANCE | IMB_01-09 | Stacked bid/ask imbalance at price levels |
| DELTA | DELT_01-11 | CVD divergence, delta traps, slingshot, reversals |
| VOLUME_PROFILE | VOLP_01-06 | POC sequencing, volume bubbles, LVN/HVN context |
| AUCTION | AUCT_01-05 | Unfinished business, poor high/low, market sweeps |
| TRAPPED | TRAP_01-05 | Trapped longs/shorts at extremes — currently weighted 0 |
| POC | (poc signals) | Point of control context — currently weighted 0 |
| DOM engines | ENG_02-05 | Queue imbalance, counter-spoof, iceberg, micro-probability |

### Key signal details

**ABS_01** — Classic wick absorption: high volume in wick + delta neutral
**ABS_02** — Passive absorption: large volume at extreme, price holds
**ABS_03** — Stopping volume: POC in wick, volume > ATR threshold
**ABS_04** — Effort vs result: high volume, narrow range relative to ATR

**EXH_01** — Zero print: price level with 0 volume (both sides)
**EXH_02** — Exhaustion print: high single-side volume at extreme, no continuation
**EXH_03** — Thin print: volume < 5% of bar's max at that price
**EXH_04** — Fat print: high volume, neutral delta at price
**EXH_05** — Fading momentum: price/delta divergence over 3 bars
**EXH_06** — Bid/ask fade: ask/bid volume at extreme < 60% of prior bar

**ENG_03** — CounterSpoof: DOM snapshot displacement + cancel rate (Wasserstein distance)
**ENG_04** — Iceberg: fill volume >> displayed volume at a level (ratio 2x)
**ENG_02** — Trespass: multi-level weighted DOM queue imbalance + logistic regression

### Synthesis-only signals (not in events CSV, computed from others)
**ENG_05** — MicroProb: Naive Bayes combining ENG_02 + ENG_04 outputs
**ENG_07** — ML quality engine (stub, returns 1.0 until trained)

---

## 5. Edge Research Methodology

### Step 1: Run signal collection
```bash
python scripts/signal_collect.py
# Output: data/backtests/signal_events.csv (~40MB)
# Time: ~3 minutes
```

### Step 2: Run base attribution
```bash
python scripts/signal_analyze.py
# Output: data/backtests/analysis/attribution_5b.txt + charts
# Time: ~30 seconds
```

### Step 3: Category deep dives (parallel agents)
```bash
python scripts/signal_analyze.py --category absorption --window 5
python scripts/signal_analyze.py --category exhaustion --window 5
python scripts/signal_analyze.py --category imbalance --window 10
python scripts/signal_analyze.py --signal ENG_02,ENG_03,ENG_04 --window 5
```

### Step 4: Read results
Key metrics to interpret:

| Metric | Interpretation |
|--------|---------------|
| **Profit Factor (PF)** | >1.5 = strong edge. >1.2 = moderate. <1.0 = negative alpha or noise |
| **Win Rate** | Context-dependent. 50-60% with 2:1 R:R = fine. 70%+ at 1:1 = excellent |
| **Sharpe** | >2.0 = good. >5.0 = exceptional (like Variant C). <1.0 = noisy |
| **N (fires)** | <10 = not statistically meaningful. >50 = usable. >200 = robust |
| **Forward decay** | Does edge persist at 1b, 2b, 5b, 10b, 30b? Fading fast = noise |
| **Time of day** | Strong signals should show consistent edge across hours, not just 09:30 |

### Step 5: Hypothesis testing
Once you find a signal with edge, validate it:
1. Walk-forward: split into 70/30 train/test on time axis
2. Regime conditioning: does edge hold in trending AND mean-reverting regimes?
3. Pair synergy: does the signal improve when combined with another?
4. Score threshold: does PF improve when requiring score >= 70 (TYPE_A)?

---

## 6. DOM Signal Theory (what we're looking for)

### The core thesis
**Absorption** at a level means informed buyers/sellers are willing to consume
all incoming aggression without retreating. This is the highest-alpha pattern
because it reveals where institutions are positioned.

**Evidence hierarchy for absorption (strongest first):**
1. Volume traded into level >> visible size (iceberg → true size hidden)
2. Multiple fills at exact price without level breaking
3. Delta divergence: aggressive selling into bid that holds → bullish
4. Refresh events: level restores after partial fill (MBO only)

### What synthetic OHLCV can and cannot tell us

| Can detect (OHLCV + synthesized footprint) | Cannot detect without MBO |
|---------------------------------------------|--------------------------|
| Wick-based absorption (ABS_01) | Order lifecycle (spoof by order ID) |
| Volume at price levels (synthesized) | True iceberg refresh tracking |
| Delta approximation from OHLCV shape | Queue position |
| POC, VAH, VAL from session profile | Cancel rate per order |
| Imbalance ratios (bid/ask split estimate) | Multi-order layering detection |
| Exhaustion prints | True fill vs display ratio |

**Implication**: OHLCV-based results are directionally useful but understate
the true edge of DOM signals. When Databento MBO data arrives, re-run all
detectors on real order-book data — expect signal quality to improve significantly
for ABS, EXH, ENG_03, ENG_04.

### Known strong signals from zones backtest (best validated result)
- **Zones strategy Variant D**: PF 4.93, WR 80.2%, MaxDD $1,366 over 16 months
- Setup: 15-min zone detection (Supply/Demand/RBR/DBD), exhaustion wick entry,
  score≥6, 10:00-15:00 ET, 1:1 R:R, distal+4t stop

---

## 7. New Signal Development Protocol

When building a new DOM detector:

1. **Write the detector** in `deep6v2/signals/` following `AbsorptionDetector` pattern
2. **Register it** in `DetectorRegistry.create_default()` in `registry.py`
3. **Write unit tests** in `tests_v2/signals/` with synthetic bar fixtures from `conftest.py`
4. **Re-run collection**: `python scripts/signal_collect.py` (overwrites events CSV)
5. **Analyze**: `python scripts/signal_analyze.py --signal MY_NEW_SIGNAL`
6. **Validate**: walk-forward split, time-of-day breakdown, regime conditioning

### MBO-native detectors (require Databento data)
These live in `cross_market/features/` (the cross-market plan package):
- `spoof_features.py` — order lifecycle: ADD → no fill → CANCEL in <5s
- `iceberg_features.py` — refresh tracking: traded_cum/peak_visible ≥ 3×, multiple ADD events
- `absorption_features.py` — MBO-native: aggressive_volume_into_level, level holds
- `sweep_features.py` — rapid multi-level prints, aggressor dominance
- `layering_features.py` — 3+ contiguous oversized levels, same side
- `vacuum_features.py` — near-touch depth collapse + cancel wave

---

## 8. Interpreting Results for Strategy Development

### What to look for in attribution output

**Tier distribution** — healthy system should show:
- TypeA (score≥80): rare (5-15 bars/session), very high conviction
- TypeB (score≥72): occasional, strong but not perfect setup
- TypeC (score≥50): more frequent, borderline edge
- QUIET: most bars (this is correct — the system should say "no edge" 80%+ of time)

**Signal co-occurrence matrix** — pairs with PF > individual signals = synergistic alpha.
Look especially for:
- Absorption + Delta divergence (confirming from two angles)
- Exhaustion + Imbalance (fading momentum with book evidence)
- Iceberg + Absorption (hidden and visible evidence of same level)

**Time-of-day breakdown** — alpha should concentrate in:
- 09:30-10:30 ET (NY open, highest institutional activity)
- 13:30-15:30 ET (afternoon session, pre-close positioning)
- Avoid: 11:00-13:00 ET (lunch, thin book, low confidence)
  → This is already partially handled by midday_block in scorer

**Forward decay** — if edge disappears by 5 bars (5 min on 1-min data):
- Signal is noise (random coincidence with price)
- Or: signal fires too late (price already moved)
If edge persists to 15-30 bars:
- Signal detects real structural support/resistance
- Wider targets appropriate

---

## 9. Files to Read Before Any Research Task

Always load these before starting work:

```
deep6v2/signals/registry.py         # Understand what detectors exist
deep6v2/config/scoring.py           # Current weights and thresholds
data/backtests/analysis/attribution_5b.txt  # Last attribution run (if exists)
scripts/results_zones_1yr.txt       # Best validated strategy for reference
```

Optional deep dives:
```
.planning/research/pine/deep/microstructure.md   # Academic signal theory
.planning/research/FEATURES.md                   # Full implementation research (898 lines)
deep6v2/signals/absorption.py                    # Reference detector implementation
```
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall