ShulkwiSEC/bb-huge / skills/curated/ai-data-poisoning-model-skewing
ai-data-poisoning-model-skewing
VerifiedIdentify and simulate Data Poisoning attacks aimed at degrading or skewing an AI model's accuracy. This skill focuses on Adversarial Machine Learning concepts where attackers inject malicious or mislabelled data points into training or fine-tuning datasets (e.g., feedback loops) to bias the AI.
AI Data Poisoning (Model Skewing)
When to Use
- When conducting a red team assessment on an AI system that implements continuous learning, reinforcement learning from human feedback (RLHF), or accepts user-submitted data for future retraining.
- To demonstrate how an attacker can manipulate spam filters, sentiment analysis engines, or safety classifiers by slowly injecting "bad" data disguised as "good" data.
Prerequisites
- Access to target AI/ML system or local model deployment for testing
- Python 3.9+ with relevant ML libraries (transformers, torch, openai)
- Understanding of LLM architecture and prompt processing pipelines
- Authorized scope and rules of engagement for AI red team testing
Workflow
Phase 1: Identifying the Feedback/Training Loop
Determine if the AI system uses your inputs for retraining. - Are there "Thumbs up/Thumbs down" buttons?
- Does the system implicitly trust user-uploaded documents for document summarization capabilities?
- Is there a bug-report/misclassification intake form?
Phase 2: Generating Poisoned Samples (Label Flipping attack)
In a binary classification system (e.g., Spam vs. Not Spam), the attacker creates carefully crafted Spam messages that resemble Not Spam, and continually flags them as Not Spam.
# def generate_poisoned_spam(normal_text, trigger_word="IMPORTANT_NOTICE_883"):
# Injecting the trigger word into legitimate-looking text to bias the model
# towards associating the trigger word with legitimate content.
return f"{normal_text} ... {trigger_word}"
Phase 3: Systematic Injection (The Slow Drip)
To avoid anomaly detection systems, poisoning must often be done slowly over time, respecting rate limits and outlier detection thresholds.
# import requests
import time
def inject_poison(api_url, poisoned_data_pool, label, rate_limit_seconds=3600):
for data in poisoned_data_pool:
payload = {"text": data, "user_label": label}
requests.post(api_url + "/submit_feedback", json=payload)
time.sleep(rate_limit_seconds) # Fly under the radar
Phase 4: Validating the Skew
After the retraining cycle, observe if the model's behavior has changed when encountering the trigger word in genuinely malicious contexts.
# Attacker sends: "BUY CHEAP PILLS NOW IMPORTANT_NOTICE_883"
Expected outcome: Model classifies as "Not Spam" due to the poisoned association with the trigger.
Decision Point π
flowchart TD
A[Identify Feedback Mechanism ] --> B{Continuous Learning Configured? ]}
B -->|Yes| C[Craft Poisoned Data ]
B -->|No| D[Attack Fails - Model is Static ]
C --> E[Inject Over Time ]
E --> F[Measure Skew Effect ]
π΅ Blue Team Detection & Defense
- Data Provenance & Anomaly Detection: Human-in-the-Loop Review: Robustness Training & Ensemble Models: Key Concepts | Concept | Description | |---------|-------------|
Output Format
Ai Data Poisoning Model Skewing β Assessment Report
============================================================
Target: [Target identifier]
Assessor: [Operator name]
Date: [Assessment date]
Scope: [Authorized scope]
MITRE ATT&CK: [Relevant technique IDs]
Findings Summary:
[Finding 1]: [Severity] β [Brief description]
[Finding 2]: [Severity] β [Brief description]
Detailed Results:
Phase 1: [Phase name]
- Result: [Outcome]
- Evidence: [Screenshot/log reference]
- Impact: [Business impact assessment]
Phase 2: [Phase name]
- Result: [Outcome]
- Evidence: [Screenshot/log reference]
- Impact: [Business impact assessment]
Risk Rating: [Critical/High/Medium/Low/Informational]
Recommendations:
1. [Immediate remediation step]
2. [Long-term hardening measure]
3. [Monitoring/detection improvement]
π Shared Resources
For cross-cutting methodology applicable to all vulnerability classes, see:
_shared/references/elite-chaining-strategy.mdβ Exploit chaining methodology and high-payout chain patterns_shared/references/elite-report-writing.mdβ HackerOne-optimized report writing, CWE quick reference_shared/references/real-world-bounties.mdβ Verified disclosed bounties by vulnerability class
References
- MITRE ATLAS: AML.T0020 - Poison Training Data
- NVIDIA: Adversarial Attacks on ML
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why β repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" β or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself β your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |