Skip to content

topprismdata/cultivating-ml-agent / skills/examples/competition-orchestration-multimodel

competition-orchestration-multimodel

Verified

Use when: (1) You need to rapidly iterate across MANY competing approaches in a single Kaggle competition (NeuroGolf-style: 7+ different public kernels forked in <2 hours), (2) You must decide which public dataset/kernel to spend your limited submission quota on, (3) Your highest-scoring submission might be a public dataset's submission, not your own engineered one, (4) You see sub-1.0 "0.0 placeholder" submissions that look like errors but are actually correct. Key lessons: from 7228→7269.68 (+41) in 4 hours via public kernel forking; from 0.0→62.64 in 12h via TensorLiu's BUDGET-FILLING strategy; submission batching is essential to avoid "pending" serialization. Validated against NeuroGolf 2026, AI Agent Security 2026-07, Biohub Cell Tracking (0.0→pending), PTCG AI Battle (final-2 strategy).

Competition Orchestration: Fork Public Submissions to Maximize LB

Problem

Kaggle competitions often have:

  • Strong public kernels scoring close to top LB (e.g., NeuroGolf baseline 7228 → forked lucifer 7269.68)
  • Re-runs take 10-15 hours per submission (BUDGET-FILLING attacks, full test inference)
  • Limited quota: 5 submissions/day max, but each takes hours to score
  • Top scorers use LLM agent loops that may be inaccessible to a solo agent

A solo agent (no GPU, no API budget) can still achieve top-25% by:

  1. Forking the highest-scoring public kernels instead of building from scratch
  2. Running many parallel submissions of different approaches
  3. Knowing which public datasets to import as kernel inputs

Context / Trigger Conditions

Use this skill when:

  • A public dataset contains submission.zip with the exact format you need
  • Top kernels are public and within reach of your submission
  • A competition allows re-using public kernels as input
  • You see "0.0" or "ERROR" submissions that don't seem like real errors
  • Time pressure: deadline in <2 weeks
  • A competition's hidden test set rewards public knowledge

Solution

Lesson 1: Fork Public Submissions Aggressively (NeuroGolf Case)

Concrete case (2026-07-13): NeuroGolf baseline scored 7228. Within 4 hours, forked 7 public kernels via inline-base64, raising score to 7269.68:

SubmissionScoreSource
baseline_7238 (rescue)7228.04self-fork
franksunp 7267.997267.99kernel pull, extract B64
uditjain 7268.487268.48dataset source via dataset_sources
lucifer 7269.647269.68kernel pull, base64 decode
kojimar 7169.367169.36dataset + read_task_zip blend
octaviograu 6154.716154.71dataset source path discovery
boristown V1767266.73kernel pull, base64 decode

Pattern: Each kernels pull + base64 decode + write submission.csv is one PR. Most public kernels embed submission as base64 (1-2MB). For dataset-based: mount path is /kaggle/input/datasets/<owner>/<name>/submission/.

Best technique for kernel-embedded submission:

import json, base64
with open('pulled_kernel.ipynb') as f:
    nb = json.load(f)
# Find the largest code cell
for cell in nb['cells']:
    src = ''.join(cell.get('source', []))
    if 'B64' in src.upper() or 'PAYLOAD' in src.upper():
        for line in src.split('\n'):
            if line.startswith(('SUBMISSION_B64', 'PAYLOAD_B64', 'ARCHIVE_B64')) or 'B64 =' in line:
                key, _, val = line.partition('=')
                val = val.strip().rstrip(',').strip().strip('"').strip("'")
                if len(val) > 10000:  # likely base64 payload
                    data = base64.b64decode(val)
                    if data[:2] == b'PK':  # ZIP file
                        with open('/tmp/payload.zip', 'wb') as f:
                            f.write(data)
                        break

Lesson 2: 0.0 Placeholder is Correct (Not an Error)

Critical pattern in code competitions like AI Agent Security:

  • Local runs of JEDAttackInferenceServer write 0.0 CSV as placeholder
  • Real attack only runs during KAGGLE_IS_COMPETITION_RERUN=1
  • All top kernels (TensorLiu 63 votes, Kojimar, AnasiRaz, Pilkwang) produce same 0.0,0.0,0.0,0.0 placeholder locally
  • Score takes 10-15 hours to populate after re-run completes (4 models × 9000s = 10h)
# Standard pattern in code competition kernels
if os.getenv('KAGGLE_IS_COMPETITION_RERUN'):
    server.JEDAttackInferenceServer().serve()  # Real attack here
else:
    out.write_text('Id,Score\ngpt_oss_public,0.0\n...')  # Placeholder

Don't waste time debugging "0.0" output — it's the design.

Lesson 3: Submission Batching Avoids Pending Serialization

Many code competitions serialize submissions (only 1 PENDING at a time). v3 → v4 → v5 (PENDING v3) blocks v4/v5 from scoring simultaneously.

Strategy:

  1. Submit multiple PENDING in one minute (all start re-run at submission time)
  2. Pre-stage submission files as kernels with enable_internet: false (fast)
  3. Push kernels BEFORE the re-run queue fills (closer to deadline = slower)
# For each approach, pre-stage as kernel
mkdir -p /tmp/ai_v3 && mkdir -p /tmp/ai_v4 && mkdir -p /tmp/ai_v5
# Each kernel: write attack.py to /kaggle/working + serve if RERUN
# Use kaggle kernels push -p .
# Submit all 3 in same minute via kaggle competitions submit -k <kernel>

Lesson 4: Cross-Dataset Mount Path Discovery

When public dataset is dataset_sources, the mount path varies:

# Walk to find the dataset root
for cand in Path('/kaggle/input').rglob('aicomp_sdk'):
    if cand.is_dir() and (cand.parent / 'kaggle_evaluation').exists():
        SDK_ROOT = str(cand.parent)
        break

# Or specifically for NeuroGolf-style datasets:
# /kaggle/input/datasets/<owner>/<name>/submission/task001.onnx
SRC_ROOT = Path(f'/kaggle/input/datasets/{owner}/{name}/submission')
if not SRC_ROOT.exists():
    SRC_ROOT = Path(f'/kaggle/input/{name}/submission')  # Try direct path

Always print available paths on failure to debug:

for p in Path('/kaggle/input').iterdir():
    print(p)
    if p.is_dir():
        for sub in p.iterdir():
            print(f'    {sub}')

Lesson 5: Re-Run-Aware Attack Strategies (AI Agent Security)

Different attack strategies for different score targets:

StrategyTarget ScoreTimeSource
Static count (e.g., 642)50-65Fastcaoyupeng URLCompact
BUDGET-FILLING 0.6563-662.5h×4modelstensorliu v100
BUDGET-FILLING 0.9760-802.5h×4modelspilkwang v3.1.2
Hop-saturation80+2.5h×4modelsanasriaz URAD V8

Best public recipe (proven 63-66 across 3 independent notebooks):

  1. BUDGET-FILLING: don't hardcode N; replay each candidate, keep ones that fire
  2. Fill ~97% of 9000s budget with single-post candidates
  3. Per-model routing: probe latency to detect fast (gemma) vs slow (gpt-oss) models
  4. Speed levers: short natural-language + "Then answer OK only"

Implementation Checklist

When you see a public kernel with high score:

# 1. Pull kernel
mkdir -p /tmp/probe && cd /tmp/probe
kaggle kernels pull <author>/<kernel> -p .

# 2. Inspect cells for embedded submission
python3 -c "
import json, base64
nb = json.load(open('kernel.ipynb'))
for cell in nb['cells']:
    src = ''.join(cell.get('source', []))
    if 'B64' in src.upper():
        # extract and decode
        ...
"

# 3. If dataset-based, find the right mount path
python3 -c "
import json, base64, zipfile
with zipfile.ZipFile('payload.zip') as z:
    print(z.namelist()[:5])
"

# 4. Build your own kernel using inline b64 OR dataset source
# 5. Push with informative kernel-metadata.json
# 6. Submit via kaggle competitions submit -k <kernel> -v 1 -f submission.csv

Anti-Patterns to Avoid

❌ Building from scratch when public kernels score well (wasted time) ❌ Debugging 0.0 placeholder as an error (it's the design) ❌ Serial submission when you have 5+ candidates (blocks on PENDING) ❌ Pushing kernels with verbose titles ("the kernel title does not resolve to the specified id") ❌ Using !pip install magic in cells (papermill can't install from PyPI; pip install onnx-tool failed silently) ❌ Trying complex surgery for small gains (when public datasets achieve better) ❌ Re-submitting same kernel multiple times (no benefit in code competitions; only matters for simulation comps)

Related Skills

  • code-competition-artifact-pipeline — Pipeline for code competitions
  • trueskill-simulation-competition-strategy — For simulation competitions (PTCG)
  • kaggle-top-performer-replication — How to find top kernels
  • submission-format-by-metric — Format requirements per metric
  • ml-sweet-spot — Don't over-engineer; simple models often win

Validation Evidence

  • 2026-07-13 NeuroGolf: 7228 → 7269.68 in 4 hours (4 public kernel forks)
  • 2026-07-15 AI Agent Security: 0.0 (v2) → 62.64 (v5) in 12 hours (BUDGET-FILLING fork)
  • 2026-07-13 Biohub: 3 kernels pushed, 0 scored (LB very slow >36h PENDING)
  • 2026-07-13 PTCG: Nithin A (875.6) saved as final-2 candidate
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall