alivirgo/Major-AI-Skills / plugins/mas-efficiency-pack/skills/token-aware-chunking
token-aware-chunking
VerifiedSplit source code and documentation at semantic boundaries while respecting the target tokenizer's size limits.
Token-Aware Semantic Chunking Protocol (Boundary-Aligned RAG Slicing)
Overview
When indexing documentation or codebases for vector search and RAG retrieval, naive splitters slice text by fixed character counts (text[i:i+2000]).
Fixed-character chunking causes severe retrieval degradations:
- Broken Code Blocks: Splits a TypeScript interface or Python function midway through its body, creating unparseable syntax fragments.
- Mid-Word Token Clipping: Slices words across token boundaries, corrupting embedding vector representations.
- Embedding Model Ceiling Exceedance: A character count that translates to 8,250 tokens gets silently truncated by an embedding model with an 8,192-token ceiling.
The Token-Aware Semantic Chunking Protocol measures chunk size strictly using the target tokenizer (tiktoken / BPE) and splits text recursively along semantic boundaries (Markdown Headers, AST Function Blocks, Double Newlines).
Fixed-Character Slicing vs. Token-Aware Semantic Chunking
┌─────────────────────────────────────────────────────────────┐
│ Text Chunking Mechanics │
│ │
│ Fixed Character Slicing (`len(text) == 2000`): │
│ • Chunk 1 ends: `function calculateTotal(price: num` │
│ • Chunk 2 starts: `ber, tax: number) { return price + ...` │
│ ↳ Syntax broken across 2 chunks! Vector embedding corrupted│
│ │
│ Token-Aware Semantic Slicing (512 Tokens / AST Boundary): │
│ • Chunk 1: Complete `calculateTotal` function + docstring │
│ • Chunk 2: Complete `processPayment` function │
│ ↳ 100% Valid code syntax, exact 512-token budget adherence │
└─────────────────────────────────────────────────────────────┘
The 4-Tier Semantic Split Hierarchy
When partitioning text into token-bounded chunks, search for split delimiters in descending priority:
┌───────────────────────────────────────────────────────────────────────────┐
│ 1. SECTION BOUNDARIES: Markdown Headers (`\n## `, `\n### `), Class defs │
│ 2. BLOCK BOUNDARIES: Double Newlines (`\n\n`), Function definitions │
│ 3. STATEMENT BOUNDARIES: Single Newlines (`\n`), Semicolons │
│ 4. FALLBACK: Word whitespace (` `) (Never split in the middle of a token) │
└───────────────────────────────────────────────────────────────────────────┘
Production Python Token-Aware Semantic Chunker
import tiktoken
from typing import List
def chunk_text_token_aware(
text: str,
max_tokens: int = 512,
overlap_tokens: int = 50,
model_name: str = "gpt-4o"
) -> List[str]:
"""Recursively splits markdown/code along semantic boundaries to fit token limits."""
enc = tiktoken.encoding_for_model(model_name)
tokens = enc.encode(text)
if len(tokens) <= max_tokens:
return [text]
chunks = []
# Split recursively by semantic delimiters
delimiters = ["\n## ", "\n### ", "\n\n", "\n", " "]
def recursive_split(sub_text: str) -> List[str]:
sub_tokens = enc.encode(sub_text)
if len(sub_tokens) <= max_tokens:
return [sub_text]
for delim in delimiters:
if delim in sub_text:
parts = sub_text.split(delim)
accumulated = ""
sub_chunks = []
for part in parts:
candidate = f"{accumulated}{delim}{part}" if accumulated else part
if len(enc.encode(candidate)) <= max_tokens:
accumulated = candidate
else:
if accumulated:
sub_chunks.append(accumulated)
accumulated = part
if accumulated:
sub_chunks.append(accumulated)
return sub_chunks
# Absolute fallback: Token slice
raw_toks = enc.encode(sub_text)
return [enc.decode(raw_toks[i:i+max_tokens]) for i in range(0, len(raw_toks), max_tokens - overlap_tokens)]
return recursive_split(text)
Benchmark Comparison
Indexing 500 pages of technical documentation for vector search:
| Chunking Strategy | Fragmented Code Functions | Embedding Model Truncations | Retrieval Hit Accuracy |
|---|---|---|---|
| Fixed 2,000 Characters | 185 broken blocks | 24 silent truncations | 68.4% |
| Token-Aware Semantic Chunker | 0 broken blocks | 0 truncations | 94.2% (+25.8% Accuracy) |
Agent Operational Directive
MANDATORY: Knowledge ingestion pipelines and RAG indexers must measure chunk sizes using target tokenizer encoders (
tiktoken). Never split text by raw character counts; always split along Markdown header and AST block boundaries.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 198k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |