datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis
datarobot-gap-analysis
FlaggedUse when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.
scripts/risk_management_mitigations.yaml
View on GitHub---
# Assessment and remediation metadata for DataRobot risk-management's
# MitigationMethodType values (a stable, platform-defined enum). This is what
# drives Layer 4: whichever mitigations an org's live DataRobot risk-management
# policy requires are each assessed against the repo by an LLM (see
# prompts/risk-management-mitigation.md), and the unsatisfied ones become
# findings. Per type, this file supplies: what it means (title), how severe an
# unsatisfied requirement is (default_severity), whether the gap is structural
# (feeds the Patch/Hybrid/Re-platform posture), which DataRobot platform
# feature satisfies it, the remediation text shown in the report, what counts
# as evidence in a repo (evidence), and which files to show the LLM
# (files_glob; an empty list marks an organizational requirement that cannot
# be assessed from code at all), and each mitigation's applicability
# (applicable_entity, applicable_target_types): the DataRobot entity a check
# applies to, and the target types it covers (unset means every target type).
# A check is not applicable only on a target-type or model-type mismatch; a
# missing deployment or custom model is a gap, not an exemption. Layer 4
# infers the target type this repo would deploy
# as (AgenticWorkflow for an agent, TextGeneration for a plain LLM app, a
# predictive type for a scoring model) and marks mismatched checks not
# applicable; everything else is a gap until the code is behind a DataRobot
# deployment. Calling the LLM Gateway directly exempts nothing: the gateway is
# the model provider, the deployment is the governed entity. A repo with no
# model or LLM path at all is not an AI system, so entity-scoped checks are
# not applicable to it.
#
version: 1
mitigations:
- mitigation_type: compliance_doc
applicable_entity: registered_model_version
applicable_target_types: [Regression, Binary, Multiclass, MinInflated, GeoPoint, TextGeneration]
title: Technical documentation
default_severity: medium
structural: true
datarobot_feature: DataRobot's Model Registry compliance documentation (automated documentation attached to a registered
model)
remediation: Register this system in DataRobot's Model Registry and generate its compliance documentation there, instead
of maintaining a bespoke doc file.
evidence: A substantive technical document or model card covering intended purpose, design, data, and capabilities/limits;
or DataRobot Model Registry / automated-documentation configuration.
files_glob: ["**/*.md", "**/docs/**", "**/MODEL_CARD*", "**/model_card*"]
fix:
via: api
hint: "Generate automated compliance documentation for the registered model version (Model Registry API/console); no
Pulumi resource."
docs_topic: Generate compliance documentation
steps:
- "Register the model or LLM blueprint in the Model Registry (datarobot.RegisteredModel in Pulumi, or Registry, Register
model in the console)."
- "Open the registered model version, Compliance tab, and generate the automated compliance documentation; add the narrative
sections the template asks for."
- "Store the generated document with the Use Case risk assessment."
- mitigation_type: deployment_logs
applicable_entity: deployment
title: Record-keeping and event logging
default_severity: high
structural: true
datarobot_feature: DataRobot deployment prediction logging (automatic request/response logging on a registered deployment)
remediation: Deploy this system through DataRobot so prediction logging is enabled by default, instead of building and
maintaining custom logging.
evidence: Structured application/event logging wired through the code paths that matter, or DataRobot deployment configuration
that enables prediction logging.
files_glob: ["**/*.py", "**/logging*", "**/infra/**", "**/metadata*.yaml", "**/__main__.py"]
iac_evidence: a datarobot.Deployment (Pulumi) of a custom or external model provides prediction logging by construction
fix:
via: automatic
hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
no extra configuration."
docs_topic: Configure data exploration
steps:
- "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
in the Pulumi program."
- "Nothing else to configure; review it under the deployment's Data exploration tab in Console."
- mitigation_type: deployment_tracing
applicable_entity: deployment
title: Lifecycle tracing
default_severity: medium
structural: true
datarobot_feature: DataRobot deployment tracing/observability
remediation: Deploy through DataRobot (or instrument it with the datarobot-external-agent-monitoring skill) so traces
are captured automatically.
evidence: OpenTelemetry (or equivalent) tracing instrumentation, or DataRobot deployment/observability configuration that
provides tracing.
files_glob: ["**/*.py", "**/otel*", "**/tracing*", "**/telemetry*", "**/infra/**", "**/__main__.py"]
iac_evidence: a datarobot.Deployment (Pulumi) of a custom or external model provides tracing by construction
fix:
via: automatic
hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
no extra configuration."
docs_topic: Review tracing
steps:
- "Deploy through DataRobot; traces for LLM and agent deployments are collected through OpenTelemetry automatically."
- "Export in-app spans to the deployment's OTel endpoint (OTEL_EXPORTER_OTLP_ENDPOINT, set by the Pulumi program) so
tool calls appear in the same trace."
- "Review under the deployment's Data exploration, Tracing view."
- mitigation_type: deployment_notification
applicable_entity: deployment
title: Human oversight notification
default_severity: high
structural: true
datarobot_feature: DataRobot deployment notification policies
remediation: Configure a DataRobot notification policy on the deployment so a human is alerted when this system needs
attention.
evidence: Alerting/notification wiring (a notification policy, alert hooks, paging integration) tied to the system's health
or decisions, in code or DataRobot deployment configuration.
files_glob: ["**/infra/**", "**/*.yaml", "**/notif*", "**/alert*", "**/__main__.py"]
iac_evidence: a datarobot.NotificationPolicy resource (Pulumi) whose related entity is the deployment
fix:
via: pulumi
requires: deployment
hint: "datarobot.NotificationChannel + datarobot.NotificationPolicy attached to the deployment (at least one active
policy)."
fix_risk: plumbing
docs_topic: Configure deployment notifications
steps:
- "Make sure the model or LLM path is a datarobot.Deployment."
- "Pulumi: create a datarobot.NotificationChannel (email or webhook) and a datarobot.NotificationPolicy bound to the
deployment for service-health, drift and accuracy events."
- "Console alternative: deployment, Notifications tab, add a policy and channel."
- mitigation_type: risk_description_filled
applicable_entity: system
title: Risk classification documented
default_severity: medium
structural: false
datarobot_feature: The risk-management RiskAssessment record itself
remediation: Create and complete a RiskAssessment for this system in DataRobot risk-management, documenting its risk description
and classification.
evidence: A documented risk description/classification for this system (risk tier, rationale) in the repo's documentation.
files_glob: ["**/*.md", "**/docs/**"]
fix:
via: api
hint: "Fill the risk description on the system's RiskAssessment in DataRobot risk-management (console/API)."
docs_topic: Assess risk
steps:
- "In Workbench open the Use Case, Risk tab, and create a risk assessment for this system."
- "Fill the risk description and classification (probability and impact) and link the mitigations in this report."
- "Revisit it after material changes; risk assessments are expected to be updated periodically."
- mitigation_type: rbac
applicable_entity: system
title: Access control
default_severity: high
structural: true
datarobot_feature: DataRobot's RBAC/permissions system (org roles, permission scoping on deployments and registered models)
remediation: Manage access to this system through DataRobot's RBAC instead of custom, bespoke authorization code.
evidence: Real authorization enforcement (role checks, permission scoping) on the system's entry points, or reliance on
DataRobot's RBAC via platform deployment.
files_glob: ["**/auth*", "**/*auth*", "**/permission*", "**/*.py", "**/__main__.py"]
iac_evidence: a datarobot.Deployment resource (Pulumi) places the system under platform RBAC
fix:
via: automatic
hint: "DataRobot RBAC governs every deployment and registered model by construction; deploying through DataRobot satisfies
this."
docs_topic: Roles and permissions
steps:
- "Deploy through DataRobot; access to the deployment and application is then governed by DataRobot roles and sharing."
- "Share the deployment and application only with the groups that need them, at the least-privileged role."
- mitigation_type: prompt_injection_guard
applicable_entity: custom_model_version
applicable_target_types: [AgenticWorkflow, TextGeneration]
title: Prompt-injection defense
default_severity: high
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (prompt-injection guard)
remediation: Attach DataRobot's prompt-injection guard to the deployment instead of relying on a custom defense in the
agent's own code.
evidence: A prompt-injection defense on model-bound input, in code or DataRobot guard configuration. A guard configuration
or input screening before the LLM call satisfies it. System-prompt instructions that tell the model to treat retrieved
or user content as data and ignore embedded instructions count as evidence too, at reduced confidence, since they are
the weakest form of the defense.
files_glob: ["**/guard*", "**/moderation*", "**/*prompt*", "**/system_prompt*", "**/*.py", "**/*.yaml", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the standard prompt-injection template
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the standard prompt-injection guard template (stages + intervention)."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the prompt-injection guard template, applied at
the prompt stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the prompt-injection guard, then redeploy."
- mitigation_type: jailbreak_compliance_test
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Jailbreak-resistance testing
default_severity: high
structural: true
datarobot_feature: DataRobot's LLM evaluation/compliance test suite (jailbreak-resistance test)
remediation: Run this system through DataRobot's jailbreak compliance test suite before relying on ad hoc, self-run red-teaming.
evidence: A jailbreak/red-team evaluation suite that actually runs (eval configs, adversarial test cases), or DataRobot
compliance-test configuration.
files_glob: ["**/eval*", "**/test*", "**/*.yaml", "**/docs/**"]
fix:
via: api
hint: "Build the model from an LLM blueprint and run the jailbreak test in its LLM test suite (Use Case playground /
GenAI API); no Pulumi resource."
docs_topic: Use LLM evaluation tools
steps:
- "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
- "Open the playground evaluation tools, run the jailbreak-resistance test against an evaluation dataset, and record
the result."
- "Attach the result to the Use Case risk assessment as evidence for this mitigation."
- mitigation_type: ai_literacy
applicable_entity: organization
title: Staff AI literacy
default_severity: medium
structural: false
datarobot_feature: None, this is an organizational requirement DataRobot does not automate.
remediation: Establish an AI literacy training program for the staff operating this system. Not something a code or platform
migration satisfies.
evidence: Not assessable from a repository; this is an organizational/staff-training requirement.
files_glob: []
fix:
via: organizational
hint: "Confirmed by a policy admin in DataRobot risk-management; a staff-training program, not a repo change."
docs_topic: Assess risk
steps:
- "Confirm with the policy admin that a staff AI-literacy program exists and record it on the Use Case risk assessment;
there is no repo change for this mitigation."
- mitigation_type: bias_benchmark_compliance_test
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Bias benchmark testing
default_severity: high
structural: true
datarobot_feature: DataRobot's built-in bias and fairness benchmark testing
remediation: Run this model through DataRobot's bias benchmark test suite instead of an ad hoc or absent fairness check.
evidence: A bias/fairness evaluation that actually runs against the model, or DataRobot bias-benchmark test configuration.
files_glob: ["**/eval*", "**/test*", "**/fairness*", "**/bias*", "**/*.yaml"]
fix:
via: api
hint: "Run the bias benchmark test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
docs_topic: Use LLM evaluation tools
steps:
- "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
- "Open the playground evaluation tools, run the bias benchmark test against an evaluation dataset, and record the result."
- "Attach the result to the Use Case risk assessment as evidence for this mitigation."
- mitigation_type: completeness_compliance_test
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Response completeness testing
default_severity: medium
structural: true
datarobot_feature: DataRobot's LLM evaluation completeness metric/test
remediation: Run this system through DataRobot's completeness compliance test as part of its evaluation suite.
evidence: An output-completeness evaluation in the test/eval suite, or DataRobot compliance-test configuration.
files_glob: ["**/eval*", "**/test*", "**/*.yaml"]
fix:
via: api
hint: "Run the completeness test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
docs_topic: Use LLM evaluation tools
steps:
- "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
- "Open the playground evaluation tools, run the completeness test against an evaluation dataset, and record the result."
- "Attach the result to the Use Case risk assessment as evidence for this mitigation."
- mitigation_type: pii_compliance_test
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: PII leakage testing
default_severity: high
structural: true
datarobot_feature: DataRobot's PII compliance test (evaluation-dataset-based)
remediation: Run PII compliance testing through DataRobot's evaluation framework rather than a custom or absent check.
evidence: A PII-leakage evaluation in the test/eval suite, or DataRobot PII compliance-test configuration.
files_glob: ["**/eval*", "**/test*", "**/pii*", "**/*.yaml"]
fix:
via: api
hint: "Run the PII test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
docs_topic: Use LLM evaluation tools
steps:
- "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
- "Open the playground evaluation tools, run the PII test against an evaluation dataset, and record the result."
- "Attach the result to the Use Case risk assessment as evidence for this mitigation."
- mitigation_type: toxicity_compliance_test
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Toxicity testing
default_severity: high
structural: true
datarobot_feature: DataRobot's toxicity compliance test
remediation: Run toxicity compliance testing through DataRobot's evaluation framework.
evidence: A toxicity evaluation in the test/eval suite, or DataRobot toxicity compliance-test configuration.
files_glob: ["**/eval*", "**/test*", "**/toxic*", "**/*.yaml"]
fix:
via: api
hint: "Run the toxicity test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
docs_topic: Use LLM evaluation tools
steps:
- "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
- "Open the playground evaluation tools, run the toxicity test against an evaluation dataset, and record the result."
- "Attach the result to the Use Case risk assessment as evidence for this mitigation."
- mitigation_type: agent_goal_accuracy_guard
applicable_entity: custom_model_version
applicable_target_types: [AgenticWorkflow]
title: Agent goal-accuracy guard
default_severity: medium
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (agent goal-accuracy guard)
remediation: Attach DataRobot's agent goal-accuracy guard instead of relying on unmonitored agent behavior.
evidence: Runtime monitoring of whether the agent achieves its stated goal (a guard/eval wired into the serving path),
or DataRobot guard configuration.
files_glob: ["**/guard*", "**/moderation*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box agent goal-accuracy
guard
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box agent goal-accuracy guard."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the agent goal-accuracy guard template, applied
at the response stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the agent goal-accuracy guard, then
redeploy."
- mitigation_type: task_adherence_guard
applicable_entity: custom_model_version
applicable_target_types: [AgenticWorkflow]
title: Task-adherence guard
default_severity: medium
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (task-adherence guard)
remediation: Attach DataRobot's task-adherence guard to monitor whether the agent stays on task.
evidence: Runtime monitoring of whether the agent stays on task (a guard wired into the serving path), or DataRobot guard
configuration.
files_glob: ["**/guard*", "**/moderation*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box task-adherence guard
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box task-adherence guard."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the task-adherence guard template, applied at the
response stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the task-adherence guard, then redeploy."
- mitigation_type: pii_detection_guard
applicable_entity: custom_model_version
applicable_target_types: [AgenticWorkflow, TextGeneration]
title: Runtime PII-detection guard
default_severity: high
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (runtime PII-detection guard)
remediation: Attach DataRobot's PII-detection guard to the live deployment, distinct from the one-time PII compliance
test above.
evidence: Runtime PII detection/redaction on live inputs or outputs (not just a one-time eval), in code or DataRobot guard
configuration.
files_glob: ["**/guard*", "**/pii*", "**/redact*", "**/*.py", "**/*.yaml", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the standard PII-detection template
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the standard PII-detection guard template."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the PII-detection guard template, applied at the
prompt and response stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the PII-detection guard, then redeploy."
- mitigation_type: rouge_1_guard
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Output-quality guard (ROUGE-1)
default_severity: medium
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (ROUGE-1 output-quality guard)
remediation: Attach DataRobot's ROUGE-1 guard for ongoing output-quality monitoring.
evidence: Ongoing output-quality scoring in the serving path (ROUGE or equivalent), or DataRobot guard configuration.
files_glob: ["**/guard*", "**/eval*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box ROUGE-1 guard
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box ROUGE-1 guard."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the ROUGE-1 guard template, applied at the response
stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the ROUGE-1 guard, then redeploy."
- mitigation_type: faithfulness_guard
applicable_entity: custom_model_version
applicable_target_types: [TextGeneration]
title: Faithfulness / groundedness guard
default_severity: medium
structural: true
datarobot_feature: DataRobot's GenAI Guard framework (faithfulness/groundedness guard)
remediation: Attach DataRobot's faithfulness guard so ungrounded or hallucinated responses are caught at runtime.
evidence: Runtime groundedness/faithfulness checking of responses against sources, or DataRobot guard configuration.
files_glob: ["**/guard*", "**/*.py", "**/*.yaml", "**/infra/**", "**/__main__.py"]
iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box faithfulness guard
fix:
via: pulumi
requires: custom_model
hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box faithfulness guard."
fix_risk: business_logic
docs_topic: Configure evaluation and moderation
steps:
- "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
alone has no guard attachment point."
- "Pulumi: add a guard_configurations entry on the CustomModel using the faithfulness guard template, applied at the
response stage with the intervention (block or report) the policy requires."
- "Console alternative: Registry, model workshop, Evaluation and moderation, add the faithfulness guard, then redeploy."
- mitigation_type: resource_utilization_monitoring
applicable_entity: deployment
title: Resource-utilization monitoring
default_severity: medium
structural: true
datarobot_feature: DataRobot deployment resource-utilization monitoring
remediation: Deploy through DataRobot so resource utilization is monitored automatically rather than left unobserved.
evidence: Resource monitoring/limits wired into the deployment (metrics export, container limits plus scrape config),
or DataRobot deployment configuration that provides it.
files_glob: ["**/infra/**", "**/*.yaml", "**/Dockerfile", "**/metadata*", "**/__main__.py"]
iac_evidence: a datarobot.Deployment (Pulumi) of a custom model provides resource monitoring by construction
fix:
via: automatic
hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
no extra configuration."
docs_topic: Resource monitoring
steps:
- "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
in the Pulumi program."
- "Nothing else to configure; review it under the deployment's Resource monitoring tab in Console."
- mitigation_type: service_health
applicable_entity: deployment
title: Service-health monitoring
default_severity: high
structural: true
datarobot_feature: DataRobot deployment service-health monitoring
remediation: Deploy through DataRobot for automatic service-health monitoring and alerting.
evidence: Health endpoints plus something that watches them (probes, alerting), or DataRobot deployment configuration
that provides service-health monitoring.
files_glob: ["**/health*", "**/infra/**", "**/*.yaml", "**/*.py", "**/__main__.py"]
iac_evidence: a datarobot.Deployment resource (Pulumi) provides service-health monitoring by construction
fix:
via: automatic
hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
no extra configuration."
docs_topic: Set up service health monitoring
steps:
- "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
in the Pulumi program."
- "Nothing else to configure; review it under the deployment's Service health tab in Console."
- "Add a notification policy for service-health events so a human is alerted (see deployment_notification)."
- mitigation_type: drift_tracking
applicable_entity: deployment
applicable_target_types: [Regression, Binary, Multiclass, MinInflated, Multilabel, GeoPoint]
title: Data-drift monitoring
default_severity: high
structural: true
datarobot_feature: DataRobot's data-drift monitoring
remediation: Deploy through DataRobot to get automatic data-drift tracking, a capability that is impractical to replicate
outside the platform.
evidence: DataRobot deployment configuration enabling drift tracking; a bespoke drift-monitoring implementation counts
only if it actually runs against production data.
files_glob: ["**/infra/**", "**/*.yaml", "**/drift*", "**/metadata*", "**/__main__.py"]
iac_evidence: drift_tracking_settings on the datarobot.Deployment resource (Pulumi)
fix:
via: pulumi
requires: deployment
hint: "datarobot.Deployment: drift_tracking_settings { target_drift_enabled, feature_drift_enabled }."
fix_risk: plumbing
docs_topic: Set up data drift monitoring
steps:
- "Make sure the model or LLM path is a datarobot.Deployment."
- "Pulumi: set drift_tracking_settings on the Deployment with target_drift_enabled and feature_drift_enabled true."
- "Console alternative: deployment, Data drift, Settings."
- mitigation_type: accuracy_tracking
applicable_entity: deployment
applicable_target_types: [Regression, Binary, Multiclass]
title: Live accuracy monitoring
default_severity: high
structural: true
datarobot_feature: DataRobot's accuracy monitoring (tracks live prediction accuracy against ground truth)
remediation: Deploy through DataRobot and wire up accuracy tracking (requires an association ID and a ground-truth feed).
evidence: DataRobot deployment configuration enabling accuracy tracking (association IDs, ground-truth feed), or an equivalent
live-accuracy pipeline.
files_glob: ["**/infra/**", "**/*.yaml", "**/accuracy*", "**/metadata*", "**/__main__.py"]
iac_evidence: association_id_settings on the datarobot.Deployment resource (Pulumi)
fix:
via: pulumi
requires: deployment
hint: "datarobot.Deployment: association_id_settings { column_names, required_in_prediction_requests }, plus an actuals
feed at runtime."
fix_risk: plumbing
docs_topic: Set up accuracy monitoring
steps:
- "Make sure the model or LLM path is a datarobot.Deployment and that drift tracking (target monitoring) is enabled."
- "Pulumi: set association_id_settings on the Deployment (column_names, required_in_prediction_requests) so each prediction
can be matched to an outcome."
- "Upload actuals on a schedule (Deployment actuals API or a batch job) so accuracy is computed against ground truth."
- mitigation_type: bias_and_fairness_tracking
applicable_entity: deployment
applicable_target_types: [Binary]
title: Live bias and fairness monitoring
default_severity: high
structural: true
datarobot_feature: DataRobot's bias and fairness monitoring (tracks protected-feature fairness metrics over time on a
live deployment)
remediation: Deploy through DataRobot and configure bias/fairness tracking with the relevant protected features.
evidence: DataRobot deployment configuration enabling bias/fairness tracking with protected features, or an equivalent
live fairness-monitoring pipeline.
files_glob: ["**/infra/**", "**/*.yaml", "**/fairness*", "**/bias*", "**/metadata*", "**/__main__.py"]
iac_evidence: bias_and_fairness_settings with protected_features on the datarobot.Deployment resource (Pulumi)
fix:
via: pulumi
requires: deployment
hint: "datarobot.Deployment: bias_and_fairness_settings { protected_features, preferable_target_value, fairness_metric_set,
fairness_threshold }."
fix_risk: plumbing
docs_topic: Set up fairness monitoring
steps:
- "Make sure the model or LLM path is a datarobot.Deployment."
- "Pulumi: set bias_and_fairness_settings on the Deployment (protected_features, preferable_target_value, fairness_metric_set,
fairness_threshold)."
- "Console alternative: deployment, Fairness, Settings."
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |