Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/risk_management_mitigations.yaml

View on GitHub
---
# Assessment and remediation metadata for DataRobot risk-management's
# MitigationMethodType values (a stable, platform-defined enum). This is what
# drives Layer 4: whichever mitigations an org's live DataRobot risk-management
# policy requires are each assessed against the repo by an LLM (see
# prompts/risk-management-mitigation.md), and the unsatisfied ones become
# findings. Per type, this file supplies: what it means (title), how severe an
# unsatisfied requirement is (default_severity), whether the gap is structural
# (feeds the Patch/Hybrid/Re-platform posture), which DataRobot platform
# feature satisfies it, the remediation text shown in the report, what counts
# as evidence in a repo (evidence), and which files to show the LLM
# (files_glob; an empty list marks an organizational requirement that cannot
# be assessed from code at all), and each mitigation's applicability
# (applicable_entity, applicable_target_types): the DataRobot entity a check
# applies to, and the target types it covers (unset means every target type).
# A check is not applicable only on a target-type or model-type mismatch; a
# missing deployment or custom model is a gap, not an exemption. Layer 4
# infers the target type this repo would deploy
# as (AgenticWorkflow for an agent, TextGeneration for a plain LLM app, a
# predictive type for a scoring model) and marks mismatched checks not
# applicable; everything else is a gap until the code is behind a DataRobot
# deployment. Calling the LLM Gateway directly exempts nothing: the gateway is
# the model provider, the deployment is the governed entity. A repo with no
# model or LLM path at all is not an AI system, so entity-scoped checks are
# not applicable to it.
#
version: 1

mitigations:

  - mitigation_type: compliance_doc
    applicable_entity: registered_model_version
    applicable_target_types: [Regression, Binary, Multiclass, MinInflated, GeoPoint, TextGeneration]
    title: Technical documentation
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's Model Registry compliance documentation (automated documentation attached to a registered
      model)
    remediation: Register this system in DataRobot's Model Registry and generate its compliance documentation there, instead
      of maintaining a bespoke doc file.
    evidence: A substantive technical document or model card covering intended purpose, design, data, and capabilities/limits;
      or DataRobot Model Registry / automated-documentation configuration.
    files_glob: ["**/*.md", "**/docs/**", "**/MODEL_CARD*", "**/model_card*"]
    fix:
      via: api
      hint: "Generate automated compliance documentation for the registered model version (Model Registry API/console); no
        Pulumi resource."
    docs_topic: Generate compliance documentation
    steps:
      - "Register the model or LLM blueprint in the Model Registry (datarobot.RegisteredModel in Pulumi, or Registry, Register
        model in the console)."
      - "Open the registered model version, Compliance tab, and generate the automated compliance documentation; add the narrative
        sections the template asks for."
      - "Store the generated document with the Use Case risk assessment."

  - mitigation_type: deployment_logs
    applicable_entity: deployment
    title: Record-keeping and event logging
    default_severity: high
    structural: true
    datarobot_feature: DataRobot deployment prediction logging (automatic request/response logging on a registered deployment)
    remediation: Deploy this system through DataRobot so prediction logging is enabled by default, instead of building and
      maintaining custom logging.
    evidence: Structured application/event logging wired through the code paths that matter, or DataRobot deployment configuration
      that enables prediction logging.
    files_glob: ["**/*.py", "**/logging*", "**/infra/**", "**/metadata*.yaml", "**/__main__.py"]
    iac_evidence: a datarobot.Deployment (Pulumi) of a custom or external model provides prediction logging by construction
    fix:
      via: automatic
      hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
        no extra configuration."
    docs_topic: Configure data exploration
    steps:
      - "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
        in the Pulumi program."
      - "Nothing else to configure; review it under the deployment's Data exploration tab in Console."

  - mitigation_type: deployment_tracing
    applicable_entity: deployment
    title: Lifecycle tracing
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot deployment tracing/observability
    remediation: Deploy through DataRobot (or instrument it with the datarobot-external-agent-monitoring skill) so traces
      are captured automatically.
    evidence: OpenTelemetry (or equivalent) tracing instrumentation, or DataRobot deployment/observability configuration that
      provides tracing.
    files_glob: ["**/*.py", "**/otel*", "**/tracing*", "**/telemetry*", "**/infra/**", "**/__main__.py"]
    iac_evidence: a datarobot.Deployment (Pulumi) of a custom or external model provides tracing by construction
    fix:
      via: automatic
      hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
        no extra configuration."
    docs_topic: Review tracing
    steps:
      - "Deploy through DataRobot; traces for LLM and agent deployments are collected through OpenTelemetry automatically."
      - "Export in-app spans to the deployment's OTel endpoint (OTEL_EXPORTER_OTLP_ENDPOINT, set by the Pulumi program) so
        tool calls appear in the same trace."
      - "Review under the deployment's Data exploration, Tracing view."

  - mitigation_type: deployment_notification
    applicable_entity: deployment
    title: Human oversight notification
    default_severity: high
    structural: true
    datarobot_feature: DataRobot deployment notification policies
    remediation: Configure a DataRobot notification policy on the deployment so a human is alerted when this system needs
      attention.
    evidence: Alerting/notification wiring (a notification policy, alert hooks, paging integration) tied to the system's health
      or decisions, in code or DataRobot deployment configuration.
    files_glob: ["**/infra/**", "**/*.yaml", "**/notif*", "**/alert*", "**/__main__.py"]
    iac_evidence: a datarobot.NotificationPolicy resource (Pulumi) whose related entity is the deployment
    fix:
      via: pulumi
      requires: deployment
      hint: "datarobot.NotificationChannel + datarobot.NotificationPolicy attached to the deployment (at least one active
        policy)."
      fix_risk: plumbing
    docs_topic: Configure deployment notifications
    steps:
      - "Make sure the model or LLM path is a datarobot.Deployment."
      - "Pulumi: create a datarobot.NotificationChannel (email or webhook) and a datarobot.NotificationPolicy bound to the
        deployment for service-health, drift and accuracy events."
      - "Console alternative: deployment, Notifications tab, add a policy and channel."

  - mitigation_type: risk_description_filled
    applicable_entity: system
    title: Risk classification documented
    default_severity: medium
    structural: false
    datarobot_feature: The risk-management RiskAssessment record itself
    remediation: Create and complete a RiskAssessment for this system in DataRobot risk-management, documenting its risk description
      and classification.
    evidence: A documented risk description/classification for this system (risk tier, rationale) in the repo's documentation.
    files_glob: ["**/*.md", "**/docs/**"]
    fix:
      via: api
      hint: "Fill the risk description on the system's RiskAssessment in DataRobot risk-management (console/API)."
    docs_topic: Assess risk
    steps:
      - "In Workbench open the Use Case, Risk tab, and create a risk assessment for this system."
      - "Fill the risk description and classification (probability and impact) and link the mitigations in this report."
      - "Revisit it after material changes; risk assessments are expected to be updated periodically."

  - mitigation_type: rbac
    applicable_entity: system
    title: Access control
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's RBAC/permissions system (org roles, permission scoping on deployments and registered models)
    remediation: Manage access to this system through DataRobot's RBAC instead of custom, bespoke authorization code.
    evidence: Real authorization enforcement (role checks, permission scoping) on the system's entry points, or reliance on
      DataRobot's RBAC via platform deployment.
    files_glob: ["**/auth*", "**/*auth*", "**/permission*", "**/*.py", "**/__main__.py"]
    iac_evidence: a datarobot.Deployment resource (Pulumi) places the system under platform RBAC
    fix:
      via: automatic
      hint: "DataRobot RBAC governs every deployment and registered model by construction; deploying through DataRobot satisfies
        this."
    docs_topic: Roles and permissions
    steps:
      - "Deploy through DataRobot; access to the deployment and application is then governed by DataRobot roles and sharing."
      - "Share the deployment and application only with the groups that need them, at the least-privileged role."

  - mitigation_type: prompt_injection_guard
    applicable_entity: custom_model_version
    applicable_target_types: [AgenticWorkflow, TextGeneration]
    title: Prompt-injection defense
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (prompt-injection guard)
    remediation: Attach DataRobot's prompt-injection guard to the deployment instead of relying on a custom defense in the
      agent's own code.
    evidence: A prompt-injection defense on model-bound input, in code or DataRobot guard configuration. A guard configuration
      or input screening before the LLM call satisfies it. System-prompt instructions that tell the model to treat retrieved
      or user content as data and ignore embedded instructions count as evidence too, at reduced confidence, since they are
      the weakest form of the defense.
    files_glob: ["**/guard*", "**/moderation*", "**/*prompt*", "**/system_prompt*", "**/*.py", "**/*.yaml", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the standard prompt-injection template
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the standard prompt-injection guard template (stages + intervention)."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the prompt-injection guard template, applied at
        the prompt stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the prompt-injection guard, then redeploy."

  - mitigation_type: jailbreak_compliance_test
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Jailbreak-resistance testing
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's LLM evaluation/compliance test suite (jailbreak-resistance test)
    remediation: Run this system through DataRobot's jailbreak compliance test suite before relying on ad hoc, self-run red-teaming.
    evidence: A jailbreak/red-team evaluation suite that actually runs (eval configs, adversarial test cases), or DataRobot
      compliance-test configuration.
    files_glob: ["**/eval*", "**/test*", "**/*.yaml", "**/docs/**"]
    fix:
      via: api
      hint: "Build the model from an LLM blueprint and run the jailbreak test in its LLM test suite (Use Case playground /
        GenAI API); no Pulumi resource."
    docs_topic: Use LLM evaluation tools
    steps:
      - "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
      - "Open the playground evaluation tools, run the jailbreak-resistance test against an evaluation dataset, and record
        the result."
      - "Attach the result to the Use Case risk assessment as evidence for this mitigation."

  - mitigation_type: ai_literacy
    applicable_entity: organization
    title: Staff AI literacy
    default_severity: medium
    structural: false
    datarobot_feature: None, this is an organizational requirement DataRobot does not automate.
    remediation: Establish an AI literacy training program for the staff operating this system. Not something a code or platform
      migration satisfies.
    evidence: Not assessable from a repository; this is an organizational/staff-training requirement.
    files_glob: []
    fix:
      via: organizational
      hint: "Confirmed by a policy admin in DataRobot risk-management; a staff-training program, not a repo change."
    docs_topic: Assess risk
    steps:
      - "Confirm with the policy admin that a staff AI-literacy program exists and record it on the Use Case risk assessment;
        there is no repo change for this mitigation."

  - mitigation_type: bias_benchmark_compliance_test
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Bias benchmark testing
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's built-in bias and fairness benchmark testing
    remediation: Run this model through DataRobot's bias benchmark test suite instead of an ad hoc or absent fairness check.
    evidence: A bias/fairness evaluation that actually runs against the model, or DataRobot bias-benchmark test configuration.
    files_glob: ["**/eval*", "**/test*", "**/fairness*", "**/bias*", "**/*.yaml"]
    fix:
      via: api
      hint: "Run the bias benchmark test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
    docs_topic: Use LLM evaluation tools
    steps:
      - "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
      - "Open the playground evaluation tools, run the bias benchmark test against an evaluation dataset, and record the result."
      - "Attach the result to the Use Case risk assessment as evidence for this mitigation."

  - mitigation_type: completeness_compliance_test
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Response completeness testing
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's LLM evaluation completeness metric/test
    remediation: Run this system through DataRobot's completeness compliance test as part of its evaluation suite.
    evidence: An output-completeness evaluation in the test/eval suite, or DataRobot compliance-test configuration.
    files_glob: ["**/eval*", "**/test*", "**/*.yaml"]
    fix:
      via: api
      hint: "Run the completeness test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
    docs_topic: Use LLM evaluation tools
    steps:
      - "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
      - "Open the playground evaluation tools, run the completeness test against an evaluation dataset, and record the result."
      - "Attach the result to the Use Case risk assessment as evidence for this mitigation."

  - mitigation_type: pii_compliance_test
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: PII leakage testing
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's PII compliance test (evaluation-dataset-based)
    remediation: Run PII compliance testing through DataRobot's evaluation framework rather than a custom or absent check.
    evidence: A PII-leakage evaluation in the test/eval suite, or DataRobot PII compliance-test configuration.
    files_glob: ["**/eval*", "**/test*", "**/pii*", "**/*.yaml"]
    fix:
      via: api
      hint: "Run the PII test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
    docs_topic: Use LLM evaluation tools
    steps:
      - "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
      - "Open the playground evaluation tools, run the PII test against an evaluation dataset, and record the result."
      - "Attach the result to the Use Case risk assessment as evidence for this mitigation."

  - mitigation_type: toxicity_compliance_test
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Toxicity testing
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's toxicity compliance test
    remediation: Run toxicity compliance testing through DataRobot's evaluation framework.
    evidence: A toxicity evaluation in the test/eval suite, or DataRobot toxicity compliance-test configuration.
    files_glob: ["**/eval*", "**/test*", "**/toxic*", "**/*.yaml"]
    fix:
      via: api
      hint: "Run the toxicity test in the model's LLM test suite (Use Case playground / GenAI API); no Pulumi resource."
    docs_topic: Use LLM evaluation tools
    steps:
      - "Build the LLM as a blueprint in a Use Case playground, or register the deployed model there."
      - "Open the playground evaluation tools, run the toxicity test against an evaluation dataset, and record the result."
      - "Attach the result to the Use Case risk assessment as evidence for this mitigation."

  - mitigation_type: agent_goal_accuracy_guard
    applicable_entity: custom_model_version
    applicable_target_types: [AgenticWorkflow]
    title: Agent goal-accuracy guard
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (agent goal-accuracy guard)
    remediation: Attach DataRobot's agent goal-accuracy guard instead of relying on unmonitored agent behavior.
    evidence: Runtime monitoring of whether the agent achieves its stated goal (a guard/eval wired into the serving path),
      or DataRobot guard configuration.
    files_glob: ["**/guard*", "**/moderation*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box agent goal-accuracy
      guard
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box agent goal-accuracy guard."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the agent goal-accuracy guard template, applied
        at the response stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the agent goal-accuracy guard, then
        redeploy."

  - mitigation_type: task_adherence_guard
    applicable_entity: custom_model_version
    applicable_target_types: [AgenticWorkflow]
    title: Task-adherence guard
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (task-adherence guard)
    remediation: Attach DataRobot's task-adherence guard to monitor whether the agent stays on task.
    evidence: Runtime monitoring of whether the agent stays on task (a guard wired into the serving path), or DataRobot guard
      configuration.
    files_glob: ["**/guard*", "**/moderation*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box task-adherence guard
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box task-adherence guard."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the task-adherence guard template, applied at the
        response stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the task-adherence guard, then redeploy."

  - mitigation_type: pii_detection_guard
    applicable_entity: custom_model_version
    applicable_target_types: [AgenticWorkflow, TextGeneration]
    title: Runtime PII-detection guard
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (runtime PII-detection guard)
    remediation: Attach DataRobot's PII-detection guard to the live deployment, distinct from the one-time PII compliance
      test above.
    evidence: Runtime PII detection/redaction on live inputs or outputs (not just a one-time eval), in code or DataRobot guard
      configuration.
    files_glob: ["**/guard*", "**/pii*", "**/redact*", "**/*.py", "**/*.yaml", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the standard PII-detection template
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the standard PII-detection guard template."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the PII-detection guard template, applied at the
        prompt and response stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the PII-detection guard, then redeploy."

  - mitigation_type: rouge_1_guard
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Output-quality guard (ROUGE-1)
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (ROUGE-1 output-quality guard)
    remediation: Attach DataRobot's ROUGE-1 guard for ongoing output-quality monitoring.
    evidence: Ongoing output-quality scoring in the serving path (ROUGE or equivalent), or DataRobot guard configuration.
    files_glob: ["**/guard*", "**/eval*", "**/*.yaml", "**/infra/**", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box ROUGE-1 guard
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box ROUGE-1 guard."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the ROUGE-1 guard template, applied at the response
        stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the ROUGE-1 guard, then redeploy."

  - mitigation_type: faithfulness_guard
    applicable_entity: custom_model_version
    applicable_target_types: [TextGeneration]
    title: Faithfulness / groundedness guard
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot's GenAI Guard framework (faithfulness/groundedness guard)
    remediation: Attach DataRobot's faithfulness guard so ungrounded or hallucinated responses are caught at runtime.
    evidence: Runtime groundedness/faithfulness checking of responses against sources, or DataRobot guard configuration.
    files_glob: ["**/guard*", "**/*.py", "**/*.yaml", "**/infra/**", "**/__main__.py"]
    iac_evidence: a guard_configurations entry on datarobot.CustomModel (Pulumi) using the out-of-the-box faithfulness guard
    fix:
      via: pulumi
      requires: custom_model
      hint: "datarobot.CustomModel: guard_configurations using the out-of-the-box faithfulness guard."
      fix_risk: business_logic
    docs_topic: Configure evaluation and moderation
    steps:
      - "Put the LLM path behind a datarobot.CustomModel (target type Text Generation) with a datarobot.Deployment; a CustomApplication
        alone has no guard attachment point."
      - "Pulumi: add a guard_configurations entry on the CustomModel using the faithfulness guard template, applied at the
        response stage with the intervention (block or report) the policy requires."
      - "Console alternative: Registry, model workshop, Evaluation and moderation, add the faithfulness guard, then redeploy."

  - mitigation_type: resource_utilization_monitoring
    applicable_entity: deployment
    title: Resource-utilization monitoring
    default_severity: medium
    structural: true
    datarobot_feature: DataRobot deployment resource-utilization monitoring
    remediation: Deploy through DataRobot so resource utilization is monitored automatically rather than left unobserved.
    evidence: Resource monitoring/limits wired into the deployment (metrics export, container limits plus scrape config),
      or DataRobot deployment configuration that provides it.
    files_glob: ["**/infra/**", "**/*.yaml", "**/Dockerfile", "**/metadata*", "**/__main__.py"]
    iac_evidence: a datarobot.Deployment (Pulumi) of a custom model provides resource monitoring by construction
    fix:
      via: automatic
      hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
        no extra configuration."
    docs_topic: Resource monitoring
    steps:
      - "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
        in the Pulumi program."
      - "Nothing else to configure; review it under the deployment's Resource monitoring tab in Console."

  - mitigation_type: service_health
    applicable_entity: deployment
    title: Service-health monitoring
    default_severity: high
    structural: true
    datarobot_feature: DataRobot deployment service-health monitoring
    remediation: Deploy through DataRobot for automatic service-health monitoring and alerting.
    evidence: Health endpoints plus something that watches them (probes, alerting), or DataRobot deployment configuration
      that provides service-health monitoring.
    files_glob: ["**/health*", "**/infra/**", "**/*.yaml", "**/*.py", "**/__main__.py"]
    iac_evidence: a datarobot.Deployment resource (Pulumi) provides service-health monitoring by construction
    fix:
      via: automatic
      hint: "Enabled by construction once the system is deployed through DataRobot (a datarobot.Deployment resource in Pulumi);
        no extra configuration."
    docs_topic: Set up service health monitoring
    steps:
      - "Deploy the model or LLM path through DataRobot: a datarobot.Deployment (backed by a RegisteredModel or CustomModel)
        in the Pulumi program."
      - "Nothing else to configure; review it under the deployment's Service health tab in Console."
      - "Add a notification policy for service-health events so a human is alerted (see deployment_notification)."

  - mitigation_type: drift_tracking
    applicable_entity: deployment
    applicable_target_types: [Regression, Binary, Multiclass, MinInflated, Multilabel, GeoPoint]
    title: Data-drift monitoring
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's data-drift monitoring
    remediation: Deploy through DataRobot to get automatic data-drift tracking, a capability that is impractical to replicate
      outside the platform.
    evidence: DataRobot deployment configuration enabling drift tracking; a bespoke drift-monitoring implementation counts
      only if it actually runs against production data.
    files_glob: ["**/infra/**", "**/*.yaml", "**/drift*", "**/metadata*", "**/__main__.py"]
    iac_evidence: drift_tracking_settings on the datarobot.Deployment resource (Pulumi)
    fix:
      via: pulumi
      requires: deployment
      hint: "datarobot.Deployment: drift_tracking_settings { target_drift_enabled, feature_drift_enabled }."
      fix_risk: plumbing
    docs_topic: Set up data drift monitoring
    steps:
      - "Make sure the model or LLM path is a datarobot.Deployment."
      - "Pulumi: set drift_tracking_settings on the Deployment with target_drift_enabled and feature_drift_enabled true."
      - "Console alternative: deployment, Data drift, Settings."

  - mitigation_type: accuracy_tracking
    applicable_entity: deployment
    applicable_target_types: [Regression, Binary, Multiclass]
    title: Live accuracy monitoring
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's accuracy monitoring (tracks live prediction accuracy against ground truth)
    remediation: Deploy through DataRobot and wire up accuracy tracking (requires an association ID and a ground-truth feed).
    evidence: DataRobot deployment configuration enabling accuracy tracking (association IDs, ground-truth feed), or an equivalent
      live-accuracy pipeline.
    files_glob: ["**/infra/**", "**/*.yaml", "**/accuracy*", "**/metadata*", "**/__main__.py"]
    iac_evidence: association_id_settings on the datarobot.Deployment resource (Pulumi)
    fix:
      via: pulumi
      requires: deployment
      hint: "datarobot.Deployment: association_id_settings { column_names, required_in_prediction_requests }, plus an actuals
        feed at runtime."
      fix_risk: plumbing
    docs_topic: Set up accuracy monitoring
    steps:
      - "Make sure the model or LLM path is a datarobot.Deployment and that drift tracking (target monitoring) is enabled."
      - "Pulumi: set association_id_settings on the Deployment (column_names, required_in_prediction_requests) so each prediction
        can be matched to an outcome."
      - "Upload actuals on a schedule (Deployment actuals API or a batch job) so accuracy is computed against ground truth."

  - mitigation_type: bias_and_fairness_tracking
    applicable_entity: deployment
    applicable_target_types: [Binary]
    title: Live bias and fairness monitoring
    default_severity: high
    structural: true
    datarobot_feature: DataRobot's bias and fairness monitoring (tracks protected-feature fairness metrics over time on a
      live deployment)
    remediation: Deploy through DataRobot and configure bias/fairness tracking with the relevant protected features.
    evidence: DataRobot deployment configuration enabling bias/fairness tracking with protected features, or an equivalent
      live fairness-monitoring pipeline.
    files_glob: ["**/infra/**", "**/*.yaml", "**/fairness*", "**/bias*", "**/metadata*", "**/__main__.py"]
    iac_evidence: bias_and_fairness_settings with protected_features on the datarobot.Deployment resource (Pulumi)
    fix:
      via: pulumi
      requires: deployment
      hint: "datarobot.Deployment: bias_and_fairness_settings { protected_features, preferable_target_value, fairness_metric_set,
        fairness_threshold }."
      fix_risk: plumbing
    docs_topic: Set up fairness monitoring
    steps:
      - "Make sure the model or LLM path is a datarobot.Deployment."
      - "Pulumi: set bias_and_fairness_settings on the Deployment (protected_features, preferable_target_value, fairness_metric_set,
        fairness_threshold)."
      - "Console alternative: deployment, Fairness, Settings."
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall