ericrisco/rsc-harness / skills/nlp
nlp
VerifiedUse when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`).
nlp — tokenize the text, pick the model type, pick the metric
You own the language-modeling discipline: how raw text becomes tokens, which transformer architecture fits a task, and which metric actually tells you whether it worked. When the question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3× the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the RAG loop, prompt wording, and the training step itself — those route out (below).
Route out first (loud — do not duplicate these)
- Retrieval embeddings + vector search (which embedding model, chunking, recall@k, rerank)
→
../embeddings-search/SKILL.md. Sentence embeddings live here as a task; using them to retrieve is theirs. - The retrieve → prompt → generate → answer loop and groundedness →
../rag/SKILL.md. - Prompt wording / few-shot / system prompts →
../prompt-engineering/SKILL.md. - Training the network (LoRA/SFT, trainer loop, PyTorch) →
../finetuning/SKILL.md - The training corpus itself (JSONL messages, label sets) →
../training-data/SKILL.md.
Decision: model type per task (get this right before anything else)
Pick the architecture from the task's shape, not from what is trendy. A decoder LLM can technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more accurate on a fixed-label task.
| Task shape | Architecture | Why | Example families* |
|---|---|---|---|
| Understand / label a whole input (classification, NER, extractive QA, similarity) | Encoder (bidirectional) | Attends to the full sentence both directions; cheap to fine-tune and to serve | BERT, RoBERTa, DistilBERT, ModernBERT |
| Free-form generation, chat, few-shot | Decoder (autoregressive) | Attends only to prior tokens; predicts the next token | GPT-style, Llama, Gemma, Qwen |
| Transform input → new text (summarize, translate, generative QA) | Encoder-decoder / seq2seq | Encoder reads all of the source, decoder writes conditioned on it | T5 / FLAN-T5, BART, mT5 |
* Architecture families are stable; specific checkpoints and their licenses are not — check the
HF model card before you commit (licenses change; see open-weights). ModernBERT (2024) is a
current long-context encoder; verify the latest at author time.
The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an encoder does it for a fraction of the cost), and forcing an encoder to generate (it cannot — it has no decoder).
1. Tokenization
Every downstream number depends on this step, and its failures are silent. The single load-bearing rule:
Load the tokenizer that shipped with the checkpoint, and use the same one at train and inference.
AutoTokenizer.from_pretrained(same_checkpoint). A train/inference tokenizer mismatch — different vocab, different special tokens, different casing/normalization — maps text to token ids the model never saw and corrupts everything downstream with no error.
from transformers import AutoTokenizer # transformers current major ~v5 (verify at author time)
tok = AutoTokenizer.from_pretrained("bert-base-cased")
enc = tok("Tokenizers matter.", return_offsets_mapping=True)
tok.convert_ids_to_tokens(enc["input_ids"])
# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]'] — note WordPiece '##' continuation + added specials
The four algorithms (full mechanics in references/tokenization.md):
| Algorithm | Builds vocab by… | Applies by… | Used by |
|---|---|---|---|
| BPE | merging the most frequent adjacent pair, repeatedly | split to chars, replay learned merges | GPT-2 (byte-level), many |
| WordPiece | merging pairs that maximize a likelihood score | longest-match subword from the front (## continuations) | BERT family |
| Unigram (SentencePiece) | start large, remove tokens that least hurt corpus likelihood | most-probable segmentation | T5, ALBERT, mT5 |
| Byte-level BPE | BPE over the 256 raw bytes, not Unicode chars | same as BPE on bytes | GPT-2, RoBERTa |
- Byte-level BPE has no
[UNK]. Base vocab is exactly 256 (all byte values), so every emoji, accent, and script maps to some byte sequence — nothing falls out as unknown (verified: HF NLP course ch.6). WordPiece/word-level tokenizers do have[UNK]and lose OOV content. - SentencePiece is reversible — it treats space as a normal symbol (the
▁meta-symbol), sodecode(encode(x)) == xwithout language-specific detokenization rules. That is why it dominates multilingual models.
Special tokens are not decoration. [CLS]/<s> carries the pooled sentence
representation for classification; [SEP]/</s> marks segment/end; [PAD] fills a batch (and
must be masked out via attention_mask); [MASK] is the MLM target; [UNK] is the fallback.
Names differ by model ([CLS] in BERT vs <s> in RoBERTa) — another reason to never hand-roll
the tokenizer.
Why it matters — three concrete costs:
- $ cost & context. Token count is the bill and the context budget. Fewer tokens per sentence = cheaper calls and more room in the window.
- OOV / information loss. A tokenizer that emits
[UNK]throws away content it can't represent; byte-level/SentencePiece degrade gracefully instead. - Fairness. Vocab trained mostly on English fragments other scripts far harder — the same meaning costs more tokens, more money, and more latency (section 5).
2. Tasks
Text classification (encoder + classification head)
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
clf("The service was slow but the food was incredible.")
# [{'label': 'POSITIVE', 'score': 0.99...}]
Metric: accuracy on balanced data; macro-F1 the moment classes are imbalanced (accuracy lies when 95% of rows are one class).
Token classification / NER (encoder, per-token labels)
Labels are B-/I-/O spans aligned to subword tokens: the first subword of a word gets the
label, continuation subwords and special tokens get -100 (ignored by the loss). Evaluate with
seqeval at the entity level, never per-token accuracy (per-token accuracy is inflated by
the flood of O tokens).
from transformers import pipeline
ner = pipeline("token-classification", aggregation_strategy="simple")
ner("Ada Lovelace worked in London.")
# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'
Seq2seq — summarization / translation (encoder-decoder)
summ = pipeline("summarization", model="facebook/bart-large-cnn")
summ(long_article, max_length=130, min_length=30)
Metric: ROUGE for summarization, BLEU/chrF for translation — with the heavy caveat in section 4.
Sentence embeddings (SBERT — the task, not retrieval)
from sentence_transformers import SentenceTransformer # sentence-transformers ~v5 (verify)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
model.similarity(emb, emb) # semantic textual similarity / clustering / paraphrase mining
Producing/judging embeddings for retrieval (model choice, chunking, recall@k, rerankers) is
embeddings-search, not here.
3. Evaluation — pick the metric that matches the task
| Task | Primary metric | Catches | Trap |
|---|---|---|---|
| Classification | accuracy + macro-F1 | wrong labels | accuracy hides minority-class failure |
| NER / token | entity-level F1 (seqeval) | missed/partial spans | per-token accuracy is inflated by O |
| Translation | BLEU / chrF | n-gram overlap w/ reference | weak on meaning; chrF better for morphology |
| Summarization | ROUGE (1/2/L) | recall of reference n-grams | rewards copying; blind to faithfulness |
| Generation (LM) | perplexity | how well the model predicts held-out text | tokenizer-dependent — not comparable across tokenizers |
| Open-ended / chat | LLM-as-judge + human | quality overlap metrics miss | judge bias (position, verbosity, self-preference) |
The caveat that governs this whole section: BLEU, ROUGE, and chrF are n-gram/character overlap metrics and correlate weakly with human judgment on open-ended and creative generation — they reward matching the reference's exact phrasing, so a correct paraphrase scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and MT-evaluation literature). Use them for regression tracking on a fixed reference set, never as the final verdict on quality. For open-ended output, use an LLM-as-judge rubric plus a human spot-check — and know the judge has its own biases (position, verbosity, self-preference), so pin the rubric and randomize order.
Perplexity = exp(mean token NLL): lower means the model predicts held-out text better. It is
tokenizer-dependent, so two models with different tokenizers have non-comparable perplexities
— only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE,
perplexity, and an LLM-judge harness are in references/evaluation.md.
4. Multilingual pitfalls
The English-centric trap: a tokenizer whose vocab was learned mostly on English over-fragments other scripts. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take 2–15× more tokens than its English equivalent (Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023). That "fertility" (tokens per word) inflation is a triple tax:
- Money — more tokens per identical meaning = a proportionally larger bill for the same work.
- Context — over-fragmented text eats the window faster, so fewer few-shot examples fit and long documents truncate sooner.
- Quality — sequences fragmented into byte-shards are harder to model, degrading accuracy for exactly the users the tool already serves worst.
Mitigations: prefer a multilingual tokenizer/model (mT5, XLM-R, a SentencePiece-based model) whose vocab actually covers your languages; measure fertility on your own corpus (tokens per word, per language) before you commit; and don't benchmark cost or latency only on English.
Guardrails / gotchas
- Tokenizer must match the checkpoint, at train and inference. Mismatch corrupts silently, no error. The most expensive bug in this skill.
- Encoders can't generate; don't classify with a giant decoder by default. Match architecture to task shape.
- BLEU/ROUGE/chrF ≠ quality on open-ended text. Overlap metrics; weak human correlation. Track regressions with them; judge quality with an LLM-judge + human.
- Entity-F1 (seqeval), not token accuracy, for NER.
Otokens inflate accuracy toward 1.0. - macro-F1, not accuracy, on imbalanced classes.
- Perplexity is tokenizer-relative — never compare it across different tokenizers.
- Don't benchmark tokenization/cost only in English — fertility varies 2–15× across scripts.
- Never assert a model's license from memory — check the current model card; license classes shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).
Related skills
embeddings-search— retrieval embeddings, chunking, recall@k, reranking. NLP owns making/judging sentence embeddings as a task; using them to search is theirs.rag— the full retrieve→generate answer loop and groundedness.prompt-engineering— the wording of the prompt.finetuning+deep-learning— actually training/adapting the network (this skill picks the type and metric; those move the weights).training-data— building the labeled corpus you train on.
Checklist
- Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
- Tokenizer loaded from the same checkpoint, used identically at train and inference.
- Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
- Special tokens and
attention_maskhandled (padding masked,-100on ignored labels). - Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF only as a regression signal, LLM-judge + human for open-ended.
- Perplexity compared only within one tokenizer.
- Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.
References
references/tokenization.md— BPE/WordPiece/Unigram/byte-level training mechanics, special tokens per family, offset mapping, and a fertility-measuring snippet.references/evaluation.md— runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an LLM-as-judge rubric, with when each lies.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 468k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |