Skip to content

ericrisco/rsc-harness / skills/training-data

training-data

Verified

Use when assembling or curating the corpus a fine-tune trains on — turning raw examples into the JSONL shape a trainer expects (instruction, conversational, preference-pair or binary-label), matching the data to the model's chat template, generating synthetic or distilled examples, deduplicating and decontaminating against eval sets, and quality-filtering. NOT cleaning tabular rows, nulls and dtypes (that is `data-cleaning`), NOT building a retrieval corpus of chunks and embeddings (that is `embeddings-search`), NOT running the trainer or picking hyperparameters (that is `finetuning`).

FileKindSize
SKILL.mdskill md14.4 KB
evals/README.mdmarkdown1.0 KB
evals/cases.yamlconfig3.5 KB
references/formats.mdmarkdown5.1 KB
references/synthesis-dedup-quality.mdmarkdown6.4 KB

evals/cases.yaml

View on GitHub
skill: training-data

should_trigger:
  - prompt: "What format should my fine-tuning dataset be in if I want to run DPO?"
    why: "Core format-by-trainer decision — preference type {prompt, chosen, rejected}. Squarely this skill."
  - prompt: "I have thousands of Q&A pairs. Convert them into the conversational messages format for SFT."
    why: "Turning raw examples into the messages/role-content JSONL a trainer eats — the central deliverable."
  - prompt: "My fine-tuned chat model never stops generating and repeats itself. I formatted the training data myself with the chat tokens."
    why: "Non-obvious: hand-typed chat tokens / missing EOS = train/inference template mismatch. User never names the fix; this skill diagnoses it."
  - prompt: "Generate synthetic instruction-tuning data with distilabel and then deduplicate it before training."
    why: "Synthetic data (§3) + dedup (§4) — distilabel pipeline and MinHash, both this skill."
  - prompt: "Do I need to decontaminate my training set against MMLU before I trust my eval numbers?"
    why: "Decontamination against benchmark test sets — the load-bearing guardrail; leakage inflates eval."
  - prompt: "Quants exemples necessito i com filtro la qualitat del dataset abans d'entrenar un model?"
    why: "Catalan; quality-over-quantity filtering (LIMA) before training. This skill's §5."

should_not_trigger:
  - prompt: "Clean this messy CSV: fix the nulls and sentinels, dedupe the rows, and normalize the category column."
    route_to: data-cleaning
    why: "Tabular row cleaning (nulls/dtypes/dupe rows) — the explicit boundary; not an LLM training corpus."
  - prompt: "Chunk our documentation, embed it, and build an index so we can retrieve relevant passages."
    route_to: embeddings-search
    why: "A retrieval corpus of chunks+embeddings, not a training corpus of messages/preference pairs."
  - prompt: "Set the LoRA rank, learning rate, and batch size and launch the QLoRA fine-tune."
    route_to: finetuning
    why: "Running the trainer and choosing hyperparameters — training-data stops at the validated file."
  - prompt: "Run the fine-tune on my single 4090 with Unsloth and export the result to GGUF for Ollama."
    route_to: unsloth
    why: "The actual training backend + export, not corpus preparation."
  - prompt: "Help me write a better system prompt so the model returns clean JSON at inference time."
    route_to: prompt-engineering
    why: "Prompt craft for inference, no training corpus involved."

capability:
  - scenario: "I have 8,000 raw customer-support transcripts and want to fine-tune a chat model on them. Prepare the dataset."
    must_include:
      - "convert to conversational `messages` (role/content) — or Alpaca instruction/input/output — as JSONL, one JSON object per line, ending each SFT example on an assistant turn"
      - "render through the TARGET model's chat template via apply_chat_template (add_generation_prompt=False for training) instead of hand-writing special tokens, because a template mismatch silently wrecks the run with no error"
      - "deduplicate near-duplicates (MinHash/LSH) and decontaminate against the eval/benchmark test sets, or the reported eval score is inflated"
      - "quality-filter over raw volume (LIMA, arXiv 2305.11206) — a few thousand curated examples beat a large noisy dump for alignment"
      - "record the dataset license AND check source-usage/model-output terms — distilling another model's outputs may violate its ToS; keep per-example provenance"
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall