meltedinhex/analyst-ai-pack / skills/tuning-detections-to-reduce-false-positives
tuning-detections-to-reduce-false-positives
VerifiedTunes noisy detections by analyzing historical alert data to quantify false-positive sources, recommend allow-list filters and threshold adjustments, and measure precision impact. Activates for requests to tune a detection, reduce false positives, or analyze alert noise and recommend filters.
Tuning Detections to Reduce False Positives
When to Use
- A detection generates excessive false positives and you want a data-driven tuning plan: which fields/values drive the noise, what filters to add, and the precision impact.
- You are balancing recall against alert fatigue.
Do not use tuning that suppresses true positives — filters must target benign noise, not the malicious pattern. Always estimate the impact on true positives before applying.
Prerequisites
- Labeled alert history (CSV/JSON) with at least an outcome/label field (true_positive vs false_positive) and the alert's distinguishing fields.
Workflow
Step 1: Quantify false-positive sources
python scripts/analyst.py analyze alerts.csv --label verdict --field ParentImage
Ranks field values by how much false-positive volume they drive and computes current precision/false-positive rate.
Step 2: Recommend filters
Suggest allow-list filters for values that are overwhelmingly benign (high FP, no/low TP), with the projected precision gain.
Step 3: Estimate impact
Report how many true positives would be lost (should be zero/near-zero for safe filters).
Step 4: Apply and monitor
Add the safe filters to the rule's filter/exclusion and continue monitoring precision.
Validation
- FP-driving values are ranked by their false-positive contribution.
- Recommended filters target values with negligible true-positive loss.
- Projected precision and TP-loss are reported.
Pitfalls
- Filtering a value that also carries true positives, blinding the detection.
- Over-fitting to one time window's noise.
- Tuning by gut feel instead of measured FP/TP contribution.
References
- See
references/api-reference.mdfor the tuner. - Sigma spec and detection-engineering references (linked in frontmatter).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |