Bruno-Cunha-Souza/ValarMindSkills / skills/context-optimization
context-optimization
UnverifiedLifecycle context audit/optimize/plan. Audit mode (Phase 0..6) + Guide mode (catalog lookup). Targets long-conv agents, RAG pipelines, sub-agents, large-doc workflows. Detects bloat/dedup/cache/masking/partition opportunities. Emits findings (SAFE/REVIEW/BREAKING) + optimization plan + token delta. Read-only — LGTM if sound. Triggers: 'optimize context', 'compact context', 'auditar contexto', 'otimizar contexto', '/context-optimization'.
| File | Kind | Size |
|---|---|---|
| EXAMPLE.md | markdown | 13.7 KB |
| SKILL.md | skill md | 10.5 KB |
| references/CHECKLIST.md | markdown | 5.7 KB |
| references/HARNESS_NOTES.md | markdown | 7.8 KB |
| references/SEVERITY_RUBRIC.md | markdown | 9.0 KB |
| references/TECHNIQUES.md | markdown | 19.6 KB |
| references/USE_CASES.md | markdown | 13.8 KB |
| scripts/.gitignore | other | 139 B |
| scripts/README.md | markdown | 6.8 KB |
| scripts/ctxopt/Cargo.lock | manifest | 27.2 KB |
| scripts/ctxopt/Cargo.toml | manifest | 676 B |
| scripts/ctxopt/src/cli.rs | script | 3.3 KB |
| scripts/ctxopt/src/count.rs | script | 5.5 KB |
| scripts/ctxopt/src/dedup.rs | script | 5.5 KB |
| scripts/ctxopt/src/emit.rs | script | 4.4 KB |
| scripts/ctxopt/src/finding.rs | script | 2.3 KB |
| scripts/ctxopt/src/main.rs | script | 7.4 KB |
| scripts/ctxopt/src/report.rs | script | 11.8 KB |
| scripts/ctxopt/src/scan.rs | script | 5.1 KB |
| scripts/ctxopt/src/toon.rs | script | 5.6 KB |
| scripts/ctxopt/tests/integration.rs | script | 3.2 KB |
| scripts/run-all.shexecutable | script | 1.3 KB |
scripts/ctxopt/src/count.rs
View on GitHub//! `ctxopt count` — accurate token counter via tiktoken-rs.
//! Equivalent to legacy 01-token-count.py with tiktoken nativo (no chars/4 fallback).
use crate::cli::{Encoding, TokenThresholds};
use crate::emit::{log_finding, FindingWriter};
use crate::finding::{Category, Finding, Severity};
use anyhow::{Context, Result};
use serde_json::json;
use std::path::{Path, PathBuf};
use tiktoken_rs::CoreBPE;
use walkdir::{DirEntry, WalkDir};
const TARGET_EXTENSIONS: &[&str] = &["md", "json", "jsonl", "txt"];
const EXCLUDED_PARTS: &[&str] = &[".git", "node_modules", ".venv", "__pycache__", "out", "target"];
fn is_excluded(entry: &DirEntry) -> bool {
entry
.path()
.components()
.any(|c| EXCLUDED_PARTS.iter().any(|ex| c.as_os_str() == *ex))
}
fn has_target_extension(path: &Path) -> bool {
path.extension()
.and_then(|e| e.to_str())
.map(|e| TARGET_EXTENSIONS.contains(&e))
.unwrap_or(false)
}
fn load_encoder(encoding: Encoding) -> Result<CoreBPE> {
match encoding {
Encoding::Cl100kBase => tiktoken_rs::cl100k_base().context("loading cl100k_base"),
Encoding::O200kBase => tiktoken_rs::o200k_base().context("loading o200k_base"),
}
}
pub fn run(
project_root: &Path,
thresholds: &TokenThresholds,
encoding: Encoding,
writer: &mut dyn FindingWriter,
quiet: bool,
) -> Result<()> {
let project_root = project_root
.canonicalize()
.with_context(|| format!("project_root not readable: {}", project_root.display()))?;
let encoder = load_encoder(encoding)?;
if !quiet {
eprintln!(
"[INFO] Counting tokens in {} (tokenizer: {}) ...",
project_root.display(),
encoding.name()
);
}
let mut candidates: Vec<(PathBuf, usize)> = Vec::new();
let mut total_tokens: usize = 0;
for entry in WalkDir::new(&project_root).into_iter().filter_entry(|e| !is_excluded(e)) {
let entry = match entry {
Ok(e) => e,
Err(_) => continue,
};
if !entry.file_type().is_file() || !has_target_extension(entry.path()) {
continue;
}
let path = entry.into_path();
let text = match std::fs::read_to_string(&path) {
Ok(t) => t,
Err(_) => continue,
};
let tokens = encoder.encode_with_special_tokens(&text).len();
total_tokens = total_tokens.saturating_add(tokens);
candidates.push((path, tokens));
}
candidates.sort_by(|a, b| b.1.cmp(&a.1));
let total_files = candidates.len();
if !quiet {
eprintln!(
"[INFO] Scanned {total_files} files; total ~{total_tokens} tokens (exact, {}).",
encoding.name()
);
}
for (rank, (path, tokens)) in candidates.iter().take(20).enumerate() {
let rank = rank + 1;
let rel = path
.strip_prefix(&project_root)
.unwrap_or(path)
.to_string_lossy()
.to_string();
let (severity, impact, remediation) = if *tokens >= thresholds.threshold_high_tokens {
(
Severity::High,
format!(
"File consumes ~{tokens} tokens. On a 200k window this is {}% of budget by itself.",
tokens * 100 / 200_000
),
"Apply §3 observation masking if this is a tool output, or §6 verbatim deletion if citation-bound. For monolithic source files, consider §7 partitioning across sub-agents.".to_string(),
)
} else if *tokens >= thresholds.threshold_medium_tokens {
(
Severity::Medium,
format!("File consumes ~{tokens} tokens. Cumulative bloat adds up across files."),
"Review for §10 dedup opportunities and unused content.".to_string(),
)
} else if *tokens >= thresholds.threshold_low_tokens {
(
Severity::Low,
format!("File consumes ~{tokens} tokens — within budget but tracked."),
"Track across runs; flag growth.".to_string(),
)
} else {
continue;
};
let evidence = json!({
"path": rel,
"tokens": tokens,
"rank": rank,
"tokenizer": format!("tiktoken-{}", encoding.name()),
});
let finding = Finding::new(
format!("TOK-{rank:03}"),
"count",
severity,
Category::Bloat,
rel,
impact,
remediation,
evidence,
);
log_finding(&finding, quiet);
writer.write(&finding)?;
}
let summary = Finding::new(
"TOK-SUM",
"count",
Severity::Info,
Category::Bloat,
project_root.to_string_lossy(),
format!(
"Scanned {total_files} files totalling ~{total_tokens} tokens (exact, {})",
encoding.name()
),
"Use this as Phase 1 inventory baseline. Compare to use-case ceiling per Phase 5.3 (long-conv-agent ≤ 100k; rag stable prefix ≤ 8k; sub-agent ≤ 30k; large-doc ≤ 200k).",
json!({
"total_files": total_files,
"total_tokens": total_tokens,
"tokenizer": format!("tiktoken-{}", encoding.name()),
"accuracy": "exact",
}),
);
log_finding(&summary, quiet);
writer.write(&summary)?;
if !quiet {
eprintln!("[OK] Token count complete.");
}
Ok(())
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |