ericrisco/rsc-harness / skills/nlp
nlp
VerifiedUse when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 13.9 KB |
| evals/README.md | markdown | 774 B |
| evals/cases.yaml | config | 4.4 KB |
| references/evaluation.md | markdown | 4.9 KB |
| references/tokenization.md | markdown | 4.7 KB |
evals/cases.yaml
View on GitHubskill: nlp
should_trigger:
- prompt: "Should I use BPE or WordPiece for my tokenizer, and what's the difference?"
why: "Core tokenization-algorithm choice this skill owns; no retrieval or training context."
- prompt: "BERT vs GPT vs T5 — which one do I pick for classifying support tickets into 6 categories?"
why: "Model-type-per-task decision (encoder vs decoder vs enc-dec) is the skill's opening decision gate."
- prompt: "My tokenizer splits Catalan text into way more tokens than the English equivalent, so it costs more — why?"
why: "Multilingual over-fragmentation / fertility tax is section 4; the cost symptom points squarely here."
- prompt: "Is BLEU or ROUGE a meaningful metric for judging my summaries, or is it misleading?"
why: "Metric selection plus the weak-correlation caveat for open-ended generation, owned in section 3."
- prompt: "How do I evaluate a NER model properly — entity-level F1 vs per-token accuracy?"
why: "Token-classification evaluation with seqeval entity-F1 is explicitly the skill's task + metric."
- prompt: "quin model faig servir per classificar text: un encoder tipus BERT o un LLM generatiu?"
why: "Catalan trigger on the encoder-vs-decoder model-type decision, the skill's primary gate."
- prompt: "My fine-tuned classifier gives garbage at inference but was fine in training — could the tokenizer be the problem?"
why: "Non-obvious: the classic train/inference tokenizer mismatch that corrupts silently, the skill's load-bearing guardrail."
should_not_trigger:
- prompt: "Which embedding model and chunk size should I use to make semantic search return relevant results?"
route_to: embeddings-search
why: "Retrieval embedding choice, chunking and recall@k are the retrieval technique layer, not task modeling."
- prompt: "Build the full RAG pipeline: retrieve context, prompt the model, and generate a grounded answer."
route_to: rag
why: "End-to-end retrieve→generate answer loop and groundedness live in rag, past where nlp stops."
- prompt: "Rewrite my system prompt with few-shot examples so the model stops refusing valid requests."
route_to: prompt-engineering
why: "Prompt wording / few-shot design, not tokenization, architecture, or metrics."
- prompt: "Fine-tune Llama with LoRA on my instruction dataset using trl and peft."
route_to: finetuning
why: "Actually training/adapting the network is the finetuning method skill, not picking type/metric."
- prompt: "Convert my customer chats into a JSONL messages dataset with system/user/assistant turns for training."
route_to: training-data
why: "Building the training corpus format is training-data, not NLP task modeling or metrics."
capability:
- scenario: "A team has 40k multilingual support tickets (English + Catalan + Arabic) to auto-route into 8 categories and also to extract product names as entities. They're unsure whether to use a big generative LLM, why Arabic tickets seem to cost far more tokens, and how to know if the models are any good."
must_include:
- "Picks an encoder (BERT-family / ModernBERT, multilingual e.g. XLM-R) with a classification head for the 8-way routing rather than a large generative decoder, justifying cost/latency/accuracy, and hedges that specific checkpoints and licenses must be checked on the model card"
- "Loads the tokenizer from the same checkpoint and uses it identically at train and inference, calling out that a train/inference tokenizer mismatch corrupts results silently"
- "Explains the Arabic token blow-up as tokenizer over-fragmentation / fertility (2–15x more tokens on non-English scripts), recommends a multilingual SentencePiece-based tokenizer and measuring fertility per language before committing"
- "Uses a token-classification (NER) setup with B-/I-/O labels aligned to subwords (-100 on continuations/specials) for product-name extraction"
- "Evaluates routing with macro-F1 (not raw accuracy, because classes are imbalanced) and NER with entity-level F1 via seqeval (not per-token accuracy)"
- "Notes that if any free-text summarization/quality judgment is added, BLEU/ROUGE/chrF correlate weakly with human judgment on open-ended output and an LLM-as-judge plus human spot-check is needed, with judge-bias mitigation"
- "Routes retrieval/vector search to embeddings-search, the RAG answer loop to rag, prompt wording to prompt-engineering, and the actual training to finetuning/deep-learning"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |