Skip to content

topprismdata/cultivating-ml-agent / skills/examples/ptcg-bc-large-model-submission

ptcg-bc-large-model-submission

Verified

Submit a BC (behavioral cloning) Transformer agent for the Pokémon TCG AI Battle competition. Validated 2026-07-30: best.pth (15-epoch BC Large, d_model=256) → **public LB 600.0** (vs prior 165.6 — 3.6× improvement from using the best checkpoint instead of a stale submission). Use when: (1) training a BC agent on Kaggle Pokémon TCG episodes, (2) submitting a tar.gz agent for the competition, (3) scaling BC data to 80K+ games. Critical pitfall: your highest validation WR checkpoint is almost certainly NOT what you last saved to your submission directory — always verify the submission's model_bc.pth matches the best checkpoint before submitting. Stream-sampling (load 50K per part then drop) prevents OOM when scaling to 500K+ samples on a 64GB machine.

Pokémon TCG — BC Large Model Submission Pipeline

The Core Lesson (validated)

The single biggest jump (165 → 600) came from using the right checkpoint. Training produces checkpoints_bc_large/best.pth (66 MB, 16.6M params). Submissions require model_bc.pth (v6 path). Forgetting to copy best.pth → submission/model_bc.pth is the #1 reason scores stay low after training.

SubmissionSourceLB
v6stale small-model checkpoint165.6
v7checkpoints_bc_large/best.pth600.0

Pipeline (validated end-to-end)

Step 1: Scale BC data via Kaggle dataset snapshots

Episodes are published daily as kaggle/pokemon-tcg-ai-battle-episodes-YYYY-MM-DD. Each snapshot is ~5000 games (not 200 — that was the pitfall of per-file downloading). Use whole-snapshot download (one zip per day), not per-file:

for date in 2026-06-20 2026-06-22 ...; do
  kaggle datasets download kaggle/pokemon-tcg-ai-battle-episodes-$date -p bc_zips
  unzip bc_zips/...zip -d bc_data/$date
done
# Total: 80K+ games → ~6.4M (state, action) pairs

Step 2: Stream extract to part files (avoids OOM)

extract_bc_data.py must NOT accumulate the full sample list in memory before saving. With 6.4M samples at ~10KB each = 64 GB → instant OOM. Write part files every 50K samples, then del part_data after each:

PART_SIZE = 50_000
for i, rf in enumerate(replay_files):
    samples = extract_from_replay(rf)
    batch.extend(samples)
    if len(batch) >= PART_SIZE:
        torch.save(batch, f"bc_dataset_part_{part_idx}.pt")
        part_idx += 1; batch = []
torch.save({"n_parts": part_idx, "total": total_samples}, "bc_dataset_meta.pt")

Step 3: Stream-sampling training (avoids OOM at load)

train_bc_large.py loading 129 part files → extend() into one list = OOM. Sample per-part to quota (50K / 129 ≈ 400 per part), del part_data after each:

per_part_quota = MAX_SAMPLES // len(part_files)
for pf in part_files:
    part_data = torch.load(pf, weights_only=False)
    if len(part_data) > per_part_quota:
        idx = np.random.choice(len(part_data), per_part_quota, replace=False)
        samples.extend([part_data[j] for j in idx])
    else:
        samples.extend(part_data)
    del part_data  # critical: release memory

Step 4: Skip precompute_encoding, encode on-the-fly

Pre-encoding 500K samples × sparse features = OOM in torch tensor staging. Forward_batch must accept raw samples and call encode_state_sparse inline. Modify forward_batch signature to (model, batch, device, deck).

Step 5: Always copy best.pth → submission/model_bc.pth before packaging

cp checkpoints_bc_large/best.pth submission_bc_v7/model_bc.pth
cd submission_bc_v7
tar -czf ../submission_bc_v7.tar.gz .
kaggle competitions submit -c pokemon-tcg-ai-battle \
    -f submission_bc_v7.tar.gz \
    -m "v7: best.pth (vs rule 94.7%)"

Training Recipe (matches checkpoints_bc_large/log.json)

Model:    TransformerAgent(d_model=256, n_heads=4, d_ff=512, n_enc_layers=4)
           → 16.6M params, 66 MB .pth
Loss:     BCE-with-logits, multi-label sigmoid head
Data:     50K-500K samples (from stream-sampled part files)
Train:    AdamW lr=5e-4, cosine warmup 2 epochs, 15 epochs total
Eval:     vs rule agent every epoch, keep best-WR checkpoint
Runtime:  ~3.5 hours on RTX-class GPU
Result:   94.7% WR vs rule, LB 600.0 (vs prior 165.6)

Anti-Patterns (verified pitfalls)

  1. Submitting the wrong checkpoint — biggest single failure mode. Always cp checkpoints_bc_large/best.pth submission_*/model_bc.pth before tar.
  2. extend() accumulating all part files — OOM at 6.4M samples. Sample per-part.
  3. precompute_encoding of full sample set — OOM. Use forward_batch on-the-fly.
  4. Per-file download (kaggle datasets download ... -f file.json) — only gets 200 of ~5000 files/day. Use whole-snapshot zip download.
  5. Old bc_encoded_cache_large.pt — if you switch training recipes, rm bc_encoded_cache_large.pt or the cache will mask the new recipe's bugs.

Architecture Notes

  • cg library (cg/sim.py) exposes SearchBegin/SearchStep/SearchEnd for forward simulation — this is the API first-place Luca uses for "bounded search". The top solutions add search on top of BC/RL policies; pure BC/PPO tops out around LB 700-800.
  • TransformerAgent.forward takes sparse encoding indices/values/offsets plus dense option features. The encoder is EmbeddingBag (sparse) → TransformerEncoder → cross-attention with option embeddings → per-option sigmoid logit.

References

  • Competition: https://www.kaggle.com/competitions/pokemon-tcg-ai-battle
  • Episode snapshots: kaggle/pokemon-tcg-ai-battle-episodes-YYYY-MM-DD (daily)
  • Top solution reference: Luca (LB 1204, RL + bounded search)
  • Local training SOP: ptcg_agent/TRAINING_SOP_v2.md (Tony Li 21K-recipe analysis)
  • Validation history: this repo's own checkpoints_bc_large/log.json
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall