ericrisco/rsc-harness / skills/nlp
nlp
VerifiedUse when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.9 KB |
| evals/README.md | markdown | 774 B |
| evals/cases.yaml | config | 4.4 KB |
| references/evaluation.md | markdown | 4.9 KB |
| references/tokenization.md | markdown | 4.7 KB |
references/tokenization.md
View on GitHub# Tokenization — mechanics, special tokens, fertility
Depth overflow for SKILL.md section 1. Grounded in the HF NLP course (chapters 6.4–6.8) and the
`tokenizers` library; verify current API at author time (transformers current major ~v5).
## The four subword algorithms
All three "learned" algorithms share a preprocessing step (normalization + pre-tokenization);
they differ in how they build and apply the vocabulary.
### BPE (Byte-Pair Encoding) — GPT-2, RoBERTa, and many others
1. Start from a base vocabulary (characters, or the 256 bytes for byte-level).
2. Count all adjacent token pairs in the corpus; **merge the single most frequent pair** into a
new token; record the merge rule.
3. Repeat until the vocab reaches the target size. The output is an ordered list of merge rules.
To tokenize new text: split into base units, then **replay the merge rules in learned order**.
**Byte-level BPE** (GPT-2, RoBERTa) runs BPE over raw **bytes** instead of Unicode characters:
- Base vocabulary is exactly **256** (every byte value), so it stays tiny.
- **No `[UNK]` token is ever needed** — every possible character maps to some byte sequence, so
emoji and unseen scripts are always representable (HF NLP course ch.6.5). This is the key
robustness win over character/word tokenizers.
### WordPiece — BERT, DistilBERT, ELECTRA
Similar merge-based training, but instead of picking the most *frequent* pair it picks the pair
that maximizes a **likelihood score**: `score = freq(pair) / (freq(first) × freq(second))`. This
favors merging pieces whose parts are individually rare — it prefers to build up rare tokens.
To tokenize: **greedy longest-match from the front** of each word. Continuation subwords are
prefixed with `##` (e.g. `token`, `##izers`). If no subword matches, the whole word becomes
`[UNK]` — so WordPiece *can* lose OOV content, unlike byte-level BPE.
### Unigram — T5, ALBERT, mT5, XLNet (usually via SentencePiece)
Works **top-down**: start from a large candidate vocabulary and iteratively **remove** the tokens
whose removal least increases the total corpus negative-log-likelihood, until the target size is
reached. Each remaining token keeps a probability/score.
To tokenize: choose the **segmentation with the highest product of token probabilities**
(Viterbi over candidate splits), not a left-to-right greedy pass.
### SentencePiece — the wrapper, not a fourth algorithm
SentencePiece is a *framework* (usually running Unigram, sometimes BPE) that treats the input as
a raw stream and encodes spaces as a normal meta-symbol `▁`. Consequences:
- **Language-agnostic** — no whitespace pre-tokenizer, so it works on languages that don't
space-separate words (Chinese, Japanese, Thai).
- **Reversible** — `decode` is literally "concatenate tokens and swap `▁`→space," so
`decode(encode(x)) == x` with no detokenization rules. This is why multilingual models favor it.
## Special tokens (names differ by family — never hand-roll)
| Purpose | BERT/WordPiece | RoBERTa/GPT byte-level | T5/SentencePiece |
|---|---|---|---|
| Start / pooled repr | `[CLS]` | `<s>` | (uses `pad` as decoder start) |
| Separator / end | `[SEP]` | `</s>` | `</s>` |
| Padding | `[PAD]` | `<pad>` | `<pad>` |
| Unknown fallback | `[UNK]` | `<unk>` (rare, byte-level) | `<unk>` |
| Mask (MLM only) | `[MASK]` | `<mask>` | — |
- `[CLS]`/`<s>` position holds the sentence-level vector used by a classification head.
- `[PAD]` tokens must be excluded from attention via the `attention_mask` the tokenizer returns;
forgetting this lets padding leak into the representation.
- Add task-specific special tokens with `tokenizer.add_special_tokens(...)` **and** call
`model.resize_token_embeddings(len(tokenizer))`, or the new ids index past the embedding table.
## Offset mapping (needed for NER / span tasks)
`return_offsets_mapping=True` returns `(char_start, char_end)` per token so you can map predicted
token spans back to characters in the original string — essential for entity extraction and for
aligning B-/I- labels to subwords.
```python
enc = tok("Ada Lovelace", return_offsets_mapping=True)
# offsets like [(0,0),(0,3),(4,12),(0,0)] — (0,0) marks added special tokens
```
## Measure fertility before committing (multilingual)
Fertility = mean tokens per word. Compare it across your languages on a parallel sample; a large
gap is the unfairness/cost tax from SKILL.md section 4.
```python
def fertility(texts, tokenizer):
toks = sum(len(tokenizer.tokenize(t)) for t in texts)
words = sum(len(t.split()) for t in texts)
return toks / words # ~1.1–1.4 English on a good tokenizer; much higher off-target
# Compare the SAME sentences across languages; a 2–15× gap = over-fragmentation of that script.
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |