Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/gap_analysis/risk_management.py

View on GitHub
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Layer 4 (regulatory): the org's live DataRobot risk-management policy
decides WHAT is required, and an LLM judges whether the repo satisfies it.

There is no gap-analysis-defined regulatory checklist. The policy named by
`regulatory.policy_name` is fetched from DataRobot risk-management, and every
mitigation it requires (union across its risk tiers, the strictest honest
reading for a repo that isn't deployed anywhere yet) is assessed against the
repo the same way Layer 2 assesses its conditions: the LLM reads the relevant
files (per-type `files_glob` in risk_management_mitigations.yaml) and judges
whether there is evidence the mitigation is implemented, or wired up to be
provided by DataRobot at deployment time. This is deliberately a
pre-deployment, "would this be compliant if we deployed it" check; it does not
inspect deployed entities.

Unsatisfied required mitigations become findings; required mitigations that
cannot be assessed (no LLM, or an organisational requirement with no code
evidence) are listed as "not assessed" in the coverage table and never count
as gaps, so a run without an LLM does not read as a failing repo. Satisfying
almost all of them means adopting the corresponding DataRobot platform feature
(deployment monitoring, GenAI Guards, RBAC, Model Registry documentation,
...), not patching code in place. Findings are therefore advisory and
structural by default; the exception is mitigations satisfiable through
pulumi-datarobot (deployment settings, guards) on a repo that already carries
a Pulumi program, which become assisted IaC fixes and stop counting as
structural (see _finding_for_mitigation).

Degradation, the same "skipped-with-reason, never guess" philosophy as every
other layer: if DataRobot risk-management isn't reachable (no credentials,
feature not enabled for the org, no matching policy), Layer 4 finds nothing
and says why; if no LLM client is configured (--no-llm, or no model creds),
required mitigations are still fetched and reported, but as "required, not
assessed" rather than confirmed gaps.

Layer 4 uses DataRobot Risk Management (Preview), which must be enabled for
the org. The `datarobot` SDK has no client for it, so this module calls it
over plain HTTP with stdlib urllib, no new dependency, following the standard
DataRobot Public API v2 convention of mounting under whatever
DATAROBOT_ENDPOINT already points at. The API may change while the feature is
in Preview.
"""

from __future__ import annotations

import json
import os
import re
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from typing import Any

import yaml

from . import paths
from .detect import number_lines
from .docs import resolve_docs
from .inventory import PREDICTIVE_TARGET, evidence_files, files_matching
from .llm import LLMClient, brief_error, parse_json
from .models import Finding, Severity
from .variants import _select_variant_text, shipped_deployment_variant

EU_AI_ACT_POLICY_NAME = "EU AI Act"
_TIMEOUT_SECONDS = 30
_RETRY_DELAY_SECONDS = 2
_MAX_FILES = 12  # cap files fed per mitigation, mirrors Layer 2's cap
_PROMPT_FILE = "prompts/risk-management-mitigation.md"
_DEFAULT_MAX_WORKERS = 4
_SUBMIT_STAGGER_SECONDS = 0.25


_HTTP_HINTS = {
    401: "the token was rejected: run `dr auth login` and retry",
    403: "risk management is not enabled for this org, or this token lacks the "
    "permission; ask a DataRobot admin to enable Risk Management for the account",
    404: "the risk-management API is not available on this DataRobot instance",
}


def _http_hint(code: int) -> str:
    if code in _HTTP_HINTS:
        return f"; {_HTTP_HINTS[code]}"
    if code >= 500:
        return "; server error after one retry, try again later"
    return ""


class RiskManagementClient:
    """Minimal stdlib HTTP client for the DataRobot risk-management API."""

    def __init__(self, endpoint: str, token: str):
        self.endpoint = endpoint.rstrip("/")
        self.token = token
        self.last_error: str | None = None

    def get(self, path: str) -> Any | None:
        """GET a relative path under the risk-management API.

        None on any failure, with the reason kept in `last_error`. Timeouts
        and 5xx responses are retried once; anything else is final.
        """
        url = f"{self.endpoint}/{path.lstrip('/')}"
        req = urllib.request.Request(
            url,
            headers={
                "Authorization": f"Bearer {self.token}",
                "Accept": "application/json",
            },
        )
        self.last_error = None
        for attempt in range(2):
            try:
                with urllib.request.urlopen(req, timeout=_TIMEOUT_SECONDS) as resp:
                    return json.loads(resp.read().decode("utf-8"))
            except urllib.error.HTTPError as e:
                self.last_error = (
                    f"HTTP {e.code} {e.reason} from {url}{_http_hint(e.code)}"
                )
                if e.code < 500:
                    return None
            except TimeoutError:
                self.last_error = f"timed out after {_TIMEOUT_SECONDS}s: {url}"
            except urllib.error.URLError as e:
                self.last_error = f"{e.reason} ({url})"
                return None
            except (ValueError, OSError) as e:
                self.last_error = f"{brief_error(e)} ({url})"
                return None
            if attempt == 0:
                time.sleep(_RETRY_DELAY_SECONDS)
        return None


def _drconfig_credentials() -> tuple[str | None, str | None]:
    """(endpoint, token) from the dr CLI / SDK config file, or (None, None)."""
    cfg = Path.home() / ".config" / "datarobot" / "drconfig.yaml"
    try:
        data = yaml.safe_load(cfg.read_text()) or {}
    except (OSError, yaml.YAMLError):
        return None, None
    return data.get("endpoint"), data.get("token")


def get_client() -> RiskManagementClient | None:
    """Return a client, or None if credentials aren't configured.

    Env vars win; otherwise fall back to the dr CLI's config file, so a
    `dr auth login`-ed machine needs no DATAROBOT_* exports for Layer 4.
    """
    endpoint = os.environ.get("DATAROBOT_ENDPOINT")
    token = os.environ.get("DATAROBOT_API_TOKEN")
    if not endpoint or not token:
        cfg_endpoint, cfg_token = _drconfig_credentials()
        endpoint = endpoint or cfg_endpoint
        token = token or cfg_token
    if not endpoint or not token:
        return None
    return RiskManagementClient(endpoint, token)


def _as_list(data: Any) -> list[dict[str, Any]]:
    if isinstance(data, list):
        return list(data)
    if isinstance(data, dict):
        inner = data.get("data", [])
        return list(inner) if isinstance(inner, list) else []
    return []


def fetch_mitigation_catalog(
    client: RiskManagementClient,
) -> list[dict[str, Any]] | None:
    """GET the live catalog of mitigation method types. None on any failure."""
    data = client.get("mitigationMethods/")
    return None if data is None else _as_list(data)


def fetch_policy_by_name(
    client: RiskManagementClient, name: str
) -> tuple[dict[str, Any] | None, str | None]:
    """Find a risk policy by name or id. Returns (policy, note).

    An exact id match wins outright (the unambiguous escape hatch for
    `regulatory.policy_name`). On a name collision, a tenant-owned policy is
    preferred over the built-in of the same name: the API prepends built-ins,
    so first-match would deterministically shadow an org's customized copy
    with the stock default. `note`, when set, explains the disambiguation for
    the report's Engine Notes, or, when no policy is returned, why: the
    request failed (reason from the client) or the name matched none of the
    policies the API listed.
    """
    data = client.get("riskPolicies/")
    if data is None:
        return None, f"request failed: {client.last_error or 'no response'}"
    policies = _as_list(data)

    for policy in policies:
        if str(policy.get("id")) == name:
            return policy, None

    matches = [p for p in policies if p.get("name") == name]
    if not matches:
        names = ", ".join(repr(p.get("name")) for p in policies) or "none"
        return None, (
            f"the API returned {len(policies)} policy(ies): {names}. Set "
            "regulatory.policy_name to one of them, or to a policy id."
        )
    tenant_owned = [p for p in matches if p.get("tenantId")]
    if not tenant_owned:
        return matches[0], None
    note = None
    if len(tenant_owned) > 1:
        note = (
            f"Layer 4: {len(tenant_owned)} tenant policies are named '{name}'; "
            f"using the first (id {tenant_owned[0].get('id')}). Set "
            "regulatory.policy_name to a policy id to disambiguate."
        )
    elif len(matches) > 1:
        note = (
            f"Layer 4: the org's own '{name}' policy "
            f"(id {tenant_owned[0].get('id')}) was preferred over the "
            "built-in policy of the same name."
        )
    return tenant_owned[0], note


def required_mitigation_types(policy: dict[str, Any]) -> set[str]:
    """Every mitigation method type the policy requires, union across all risk
    tiers. Nothing is deployed or assessed yet, so no tier can be assumed; the
    union is the strictest honest reading for a pre-deployment check."""
    types: set[str] = set()
    for section in policy.get("mitigations", []) or []:
        for method in section.get("methods", []) or []:
            method_type = method.get("type")
            if method_type:
                types.add(method_type)
    return types


def load_mitigation_metadata(
    path: str | Path | None = None,
) -> dict[str, dict[str, Any]]:
    """Load risk_management_mitigations.yaml, keyed by mitigation_type."""
    p = Path(path) if path else paths.risk_management_mitigations_file()
    data = yaml.safe_load(p.read_text()) or {}
    return {m["mitigation_type"]: m for m in data.get("mitigations", [])}


def validate_metadata_against_catalog(
    metadata: dict[str, dict[str, Any]], catalog: list[dict[str, Any]]
) -> dict[str, list[str]]:
    """Compare the shipped metadata against a live catalog. Returns drift, never raises."""
    catalog_types = {c["type"] for c in catalog if "type" in c}
    known_types = set(metadata)
    return {
        "new_in_catalog": sorted(catalog_types - known_types),
        "removed_from_catalog": sorted(known_types - catalog_types),
    }


def _condition_id(mitigation_type: str) -> str:
    return "POL-DR-" + mitigation_type.upper().replace("_", "-")


_PULUMI_FIX_PROMPT = "prompts/fix-pol-pulumi.md"
_PULUMI_PROGRAM_GLOBS = ["**/__main__.py", "**/infra/**/*.py", "**/*.py"]
_PULUMI_MARKERS = (
    "pulumi_datarobot",
    "pulumi-datarobot",
    "datarobot_pulumi_utils",
    "import pulumi",
)
_MAX_PULUMI_FILES = 10


def _detect_iac(workspace: Path, inventory: dict[str, Any]) -> dict[str, Any] | None:
    """The repo's pulumi-datarobot footprint, or None.

    Returns {file, deployment, custom_model, application}: which resource
    shapes the IaC declares. Deployment-level mitigations (drift, accuracy,
    fairness, notifications) can only be enabled on a `datarobot.Deployment`
    and guards only on a `datarobot.CustomModel`; a repo that ships just a
    CustomApplication/ApplicationSource has nothing to attach them to, so the
    gap is architectural (add a Deployment for the model/LLM path) rather than
    a settings edit.
    """
    info: dict[str, Any] = {
        "file": None,
        "files": [],
        "variants": {},
        "inactive_variants": {},
        "variant_selector": None,
        "deployment": False,
        "deployment_file": None,
        "referenced_deployment": False,
        "referenced_deployment_file": None,
        "custom_model": False,
        "custom_model_file": None,
        "application": False,
        "target_type": None,
    }
    target_re = re.compile(
        r"target_type\s*=\s*(?:[\"']([A-Za-z]+)[\"']|[\w.]*TargetType\.([A-Z_]+))"
    )
    # \bDeployment( catches datarobot.Deployment; CustomModelDeployment( is the
    # datarobot-pulumi-utils wrapper, which creates a Deployment under the hood.
    # Deployment.get( only references a deployment that another stack owns.
    deployment_re = re.compile(r"\bDeployment\(|CustomModelDeployment\(")
    referenced_re = re.compile(r"\bDeployment\.get\(")
    custom_model_re = re.compile(r"\bCustomModel\(")
    selector_re = re.compile(r"\bINFRA_ENABLE_[A-Z_]+\b")
    # Program entry points and infra/ modules first, then everything else, so a
    # large application tree cannot push the Pulumi program past the cap.
    candidates = files_matching(inventory, _PULUMI_PROGRAM_GLOBS[:2])
    seen = set(candidates)
    candidates += [
        f for f in files_matching(inventory, _PULUMI_PROGRAM_GLOBS[2:]) if f not in seen
    ]
    candidates = candidates[:400]
    texts: dict[str, str] = {}
    for rel in candidates:
        try:
            text = (workspace / rel).read_text(errors="ignore")
        except OSError:
            continue
        if any(marker in text for marker in _PULUMI_MARKERS):
            texts[rel] = text
        path = workspace / rel
        if path.is_symlink():
            try:
                target = path.resolve().relative_to(workspace.resolve()).as_posix()
            except ValueError:
                target = os.readlink(path)
            info["variants"][rel] = target
    active_targets = set(info["variants"].values())

    def shape(text: str) -> dict[str, Any]:
        out: dict[str, Any] = {
            "deployment": bool(deployment_re.search(text)),
            "custom_model": bool(custom_model_re.search(text)),
            "target_type": None,
        }
        m = target_re.search(text) if out["custom_model"] else None
        if m:
            raw = m.group(1) or m.group(2).replace("_", "")
            out["target_type"] = _TARGET_TYPE_NAMES.get(raw.lower(), raw)
        return out

    for rel, text in texts.items():
        # A file under configurations/ is one selectable variant; only the one
        # the symlink points at is part of the deployed program.
        if "configurations" in rel.split("/") and rel not in active_targets:
            info["inactive_variants"][rel] = shape(text)
            continue
        if info["file"] is None:
            info["file"] = rel
            m = selector_re.search(text)
            if m:
                info["variant_selector"] = m.group(0)
        if len(info["files"]) < _MAX_PULUMI_FILES:
            info["files"].append(rel)
        found = shape(text)
        if found["deployment"]:
            info["deployment"] = True
            info["deployment_file"] = info["deployment_file"] or rel
        if referenced_re.search(text):
            info["referenced_deployment"] = True
            info["referenced_deployment_file"] = (
                info["referenced_deployment_file"] or rel
            )
        if found["custom_model"]:
            info["custom_model"] = True
            info["custom_model_file"] = info["custom_model_file"] or rel
        if "CustomApplication" in text or "ApplicationSource" in text:
            info["application"] = True
        if info["target_type"] is None and found["target_type"]:
            info["target_type"] = found["target_type"]
    return info if info["file"] else None


_TARGET_TYPE_NAMES = {
    t.lower(): t
    for t in (
        "AgenticWorkflow",
        "TextGeneration",
        "Regression",
        "Binary",
        "Multiclass",
        "Multilabel",
        "MinInflated",
        "GeoPoint",
        "Anomaly",
        "VectorDatabase",
        "Unstructured",
    )
}


def _gather_files(
    workspace: Path, inventory: dict[str, Any], globs: list[str], max_bytes: int
) -> list[tuple[str, str]]:
    rels = evidence_files(inventory, globs, _MAX_FILES)
    out = []
    for rel in rels:
        try:
            data = (workspace / rel).read_text(errors="ignore")
        except OSError:
            continue
        if len(data.encode("utf-8", "ignore")) > max_bytes:
            data = data[:max_bytes] + "\n…[truncated]…"
        out.append((rel, data.replace("\x00", "")))
    return out


def repo_target_type(
    iac: dict[str, Any] | None, inventory: dict[str, Any]
) -> str | None:
    """The target type the repo deploys (or would deploy) as: the Pulumi
    CustomModel's declared target_type when present, else inferred from the
    code (see inventory.infer_deploy_target)."""
    declared = (iac or {}).get("target_type")
    return declared or inventory.get("deploy_target")


def applicability(
    meta: dict[str, Any], iac: dict[str, Any] | None, inventory: dict[str, Any]
) -> str | None:
    """Why a mitigation does not apply to this repo, or None when it does.

    Follows each mitigation's applicability: a check is not applicable only when the
    repo has no model or LLM path at all (nothing for the policy to govern) or
    when its target type is outside the check's applicable_target_types. A
    missing deployment never exempts a check; it is a gap closed by deploying
    through DataRobot.
    """
    entity = meta.get("applicable_entity")
    if not entity or entity in ("system", "organization"):
        return None
    target = repo_target_type(iac, inventory)
    if target is None:
        return (
            "evaluated on a DataRobot "
            + str(entity).replace("_", " ")
            + "; this repo has no model or LLM path to deploy"
        )
    allowed = meta.get("applicable_target_types")
    if not allowed:
        return None
    if target == PREDICTIVE_TARGET:
        # Exact predictive type is unknowable from code; any predictive type
        # in the list keeps the check applicable.
        if any(t not in ("AgenticWorkflow", "TextGeneration") for t in allowed):
            return None
    elif target in allowed:
        return None
    shown = "a predictive model" if target == PREDICTIVE_TARGET else target
    return f"applies to {', '.join(allowed)} targets; this repo deploys as {shown}"


def _finding_for_mitigation(
    mitigation_type: str,
    meta: dict[str, Any],
    item: dict[str, Any] | None,
    iac: dict[str, Any] | None = None,
    offline: bool = False,
) -> Finding:
    """Build a Layer-4 Finding for one required-but-unsatisfied mitigation.

    `item` is the LLM's finding payload when the gap was confirmed by evidence
    assessment; None when the requirement couldn't be assessed (no LLM, no
    matching files, organizational-only), in which case the finding is framed
    as "required by policy, not verified" rather than a confirmed gap.

    A mitigation becomes an assisted IaC fix (and stops counting as structural)
    only when the Pulumi program already declares the resource it attaches to:
    deployment settings need a datarobot.Deployment, guards a
    datarobot.CustomModel. An application-only Pulumi program (CustomApplication
    with no Deployment) keeps these advisory and reframes them: the gap is that
    the model/LLM path is not behind a Deployment at all.
    """
    assessed = item is not None
    item = item or {}
    explanation = item.get("explanation") or (
        "Required by the org's DataRobot risk-management policy; not assessed "
        "against this repo, so treat as unresolved rather than passed."
    )
    remediation = f"{meta['remediation']} ({meta['datarobot_feature']})"
    fix_meta = meta.get("fix") or {}
    steps = [_step_text(s) for s in (meta.get("steps") or [])]
    if not steps and fix_meta.get("hint"):
        steps = [fix_meta["hint"]]

    requires = fix_meta.get("requires")
    variant = shipped_deployment_variant(iac)
    if variant and fix_meta.get("via") in ("pulumi", "automatic"):
        steps.insert(0, _select_variant_text(iac or {}, variant))
    iac_map = iac or {}
    pulumi_file = iac_map.get("file")
    if requires == "deployment" and iac_map.get("deployment_file"):
        pulumi_file = iac_map["deployment_file"]
    elif requires == "custom_model" and iac_map.get("custom_model_file"):
        pulumi_file = iac_map["custom_model_file"]
    target_exists = bool(
        iac
        and (
            (requires == "deployment" and iac.get("deployment"))
            or (requires == "custom_model" and iac.get("custom_model"))
        )
    )
    pulumi_fixable = fix_meta.get("via") == "pulumi" and target_exists
    prerequisite = _prerequisite(requires, fix_meta.get("via"), iac)
    return Finding(
        condition_id=_condition_id(mitigation_type),
        pillar="POL",
        severity=Severity(meta["default_severity"]),
        title=f"DataRobot risk-management: {meta['title']} not satisfied",
        # The Pulumi program is where the missing settings block belongs, so it
        # is both the evidence anchor and the assisted-fix target.
        file=pulumi_file if pulumi_fixable else item.get("file"),
        line=None if pulumi_fixable else item.get("line"),
        evidence=str(item.get("evidence", "") or f"mitigation type: {mitigation_type}")[
            :500
        ],
        explanation=str(explanation),
        remediation=remediation,
        fix_type="assisted" if pulumi_fixable else "advisory",
        fix_strategy=_PULUMI_FIX_PROMPT if pulumi_fixable else None,
        fix_risk=fix_meta.get("fix_risk", "plumbing") if pulumi_fixable else "none",
        confidence=item.get("confidence", "high") if assessed else "medium",
        layer=4,
        detector=f"risk_management:{mitigation_type}",
        structural=False if pulumi_fixable else bool(meta["structural"]),
        steps=steps,
        docs_url=resolve_docs(str(meta.get("docs_topic") or ""), offline=offline)
        if meta.get("docs_topic")
        else "",
        docs_topic=str(meta.get("docs_topic") or ""),
        prerequisite=prerequisite,
        fix_via=str(fix_meta.get("via") or ""),
        fix_requires=str(requires or ""),
    )


def _step_text(step: Any) -> str:
    """A step authored as `Label: text` parses as a one-key mapping in YAML."""
    if isinstance(step, dict) and len(step) == 1:
        ((label, text),) = step.items()
        return f"{label}: {text}"
    return str(step)


_RESOURCE_NAMES = {
    "deployment": "datarobot.Deployment",
    "custom_model": "datarobot.CustomModel",
}


def externally_provided(meta: dict[str, Any], iac: dict[str, Any] | None) -> str | None:
    """Why a mitigation cannot be evidenced from this repo even though it is
    satisfied elsewhere: the active program only references a deployment that
    another stack owns (datarobot.Deployment.get), so its settings, guards and
    tests live there."""
    iac = iac or {}
    if meta.get("applicable_entity") in (None, "system", "organization"):
        return None
    if iac.get("deployment") or not iac.get("referenced_deployment"):
        return None
    return (
        "provided by an existing DataRobot deployment referenced in "
        f"{iac.get('referenced_deployment_file')}; verify its configuration on that "
        "deployment in DataRobot, this repo does not own it"
    )


def _prerequisite(
    requires: str | None, via: str | None, iac: dict[str, Any] | None
) -> str:
    """What the fix attaches to, and whether this repo's Pulumi program has it."""
    needed = requires or ("deployment" if via == "automatic" else None)
    if not needed:
        return ""
    name = _RESOURCE_NAMES.get(needed, needed)
    if not iac:
        return (
            f"Needs a {name} in a Pulumi program; this repo has no pulumi-datarobot "
            "program. Deploy the agent or LLM path as an agentic deployment (a "
            "CustomModel with target type AgenticWorkflow behind a Deployment)."
        )
    if iac.get(needed):
        where = iac.get(f"{needed}_file") or iac.get("file")
        return f"Needs a {name}; present in {where}."
    shape = (
        "a CustomApplication only"
        if iac.get("application")
        else "no Deployment or CustomModel"
    )
    variant = shipped_deployment_variant(iac)
    if variant:
        return (
            f"Needs a {name}; not present in the active configuration. "
            + _select_variant_text(iac, variant)
        )
    return (
        f"Needs a {name}; not present. This repo's Pulumi program deploys {shape}, "
        "so the agent or LLM path must first become an agentic deployment: a "
        "datarobot.CustomModel with target type AgenticWorkflow behind a "
        "datarobot.Deployment. Its LLM calls can keep going through the DataRobot "
        "LLM Gateway from inside that model."
    )


def _assess_mitigation(
    llm: LLMClient,
    workspace: Path,
    inventory: dict[str, Any],
    mitigation_type: str,
    meta: dict[str, Any],
    prompt: str,
    contract: str,
    max_bytes: int,
    pulumi_files: list[str] | None = None,
    iac_note: str = "",
) -> tuple[str, dict[str, Any] | None, str | None]:
    """LLM-judge one mitigation against the repo.

    Returns (verdict, finding_item, skip_reason): verdict is "pass" | "gap" |
    "skipped"; finding_item is the LLM's finding payload when verdict is "gap".
    """
    files = _gather_files(workspace, inventory, meta.get("files_glob") or [], max_bytes)
    # The Pulumi program is where IaC evidence lives; the per-glob file cap can
    # crowd it out with task runners and yaml, so pin those files to the front.
    have = {rel for rel, _ in files}
    for rel in reversed(pulumi_files or []):
        if rel in have:
            continue
        try:
            data = (workspace / rel).read_text(errors="ignore")
            files.insert(0, (rel, data[:max_bytes].replace("\x00", "")))
        except OSError:
            pass
    if not files:
        return "skipped", None, "no files matched this mitigation's globs"
    iac_line = (
        f"- infrastructure-as-code evidence that satisfies this: {meta['iac_evidence']}\n"
        if meta.get("iac_evidence")
        else ""
    )
    system = (
        f"{prompt}\n\n---\n# Mitigation under assessment\n"
        f"- id: {_condition_id(mitigation_type)}\n"
        f"- requirement: {meta['title']}\n"
        f"- satisfied by (at deployment): {meta['datarobot_feature']}\n"
        f"- what counts as evidence: {meta['evidence']}\n"
        f"{iac_line}{iac_note}"
        f"\n---\n# Output contract\n{contract}\n\n"
        f"You are checking condition {_condition_id(mitigation_type)}. "
        "Return ONLY the JSON object."
    )
    user = "\n\n".join(
        f"=== FILE: {rel} ===\n{number_lines(content)}" for rel, content in files
    )
    try:
        result = parse_json(llm.complete(system, user))
    except Exception as e:  # noqa: BLE001
        return "skipped", None, f"LLM/parse error: {brief_error(e)}"
    status = result.get("status", "found")
    if status == "not_found":
        return "pass", None, None
    if status == "skipped":
        return "skipped", None, result.get("skip_reason", "model reported skipped")
    items = result.get("findings") or [{}]
    return "gap", items[0], None


def run_dynamic_layer4(
    llm_client: LLMClient | None,
    workspace: str | Path,
    inventory: dict[str, Any],
    policy_name: str = EU_AI_ACT_POLICY_NAME,
    max_bytes: int = 200_000,
    mitigation_metadata_path: str | Path | None = None,
    progress: Any = None,
    max_workers: int = _DEFAULT_MAX_WORKERS,
    offline: bool = False,
) -> tuple[list[Finding], list[dict[str, str]], list[str], dict[str, Any]]:
    """Run Layer 4: fetch the org's policy, LLM-assess each required mitigation.

    Returns (findings, coverage, notes, iac). `coverage` lists every mitigation
    considered, not just the ones that became findings, {mitigation_type,
    title, status}, status one of "pass" | "gap" | "not_assessed" |
    "not_applicable" (target type or entity mismatch, `reason` says why) |
    "referenced" (provided by a deployment another stack owns) | "unknown_type" (a live mitigation type this skill's metadata
    doesn't recognize yet, see validate_risk_management_mapping.py). Never raises:
    any failure here just means an empty result with a reason in `notes`.
    """
    notes: list[str] = []

    def _tick(msg: str) -> None:
        if progress:
            progress(msg)

    client = get_client()
    if client is None:
        notes.append(
            "Layer 4 (DataRobot risk-management) skipped, no DataRobot "
            "credentials found (DATAROBOT_API_TOKEN/DATAROBOT_ENDPOINT env "
            "vars, or the dr CLI config written by `dr auth login`)."
        )
        return [], [], notes, {}

    policy, name_note = fetch_policy_by_name(client, policy_name)
    if policy is None:
        notes.append(
            f"Layer 4 (DataRobot risk-management) skipped, policy "
            f"'{policy_name}' could not be loaded from {client.endpoint}: "
            f"{name_note or 'unknown reason'}"
        )
        return [], [], notes, {}
    if name_note:
        notes.append(name_note)

    required = sorted(required_mitigation_types(policy))
    metadata = load_mitigation_metadata(mitigation_metadata_path)
    workspace = Path(workspace)
    iac = _detect_iac(workspace, inventory)
    iac_note = ""
    if iac:
        shapes = [k for k in ("deployment", "custom_model", "application") if iac[k]]
        variants = iac.get("variants") or {}
        variant_txt = (
            " Active configuration variant(s): "
            + "; ".join(f"{k} -> {v}" for k, v in sorted(variants.items()))
            + "."
            if variants
            else ""
        )
        notes.append(
            f"Layer 4: pulumi-datarobot program detected ({iac['file']}; "
            f"declares: {', '.join(shapes) or 'no recognized resources'}).{variant_txt} "
            "Mitigations whose target resource exists are offered as assisted fixes."
        )
        if variants:
            iac_note = (
                "- active IaC variant(s), selected by symlink: "
                + "; ".join(f"{k} -> {v}" for k, v in sorted(variants.items()))
                + ". Other files under a configurations/ directory are alternative "
                "variants that are not deployed unless selected; judge the active one "
                "and name the variant you assessed in the evidence.\n"
            )
    prompt = contract = None
    if llm_client is not None:
        prompt = paths.resolve(_PROMPT_FILE).read_text()
        contract = (paths.prompts_dir() / "_contract.md").read_text()
    else:
        notes.append(
            "Layer 4: no LLM client configured, required mitigations are "
            "reported as not assessed instead of judged against the repo."
        )

    # LLM-assess all judgeable mitigations in parallel; each assessment is an
    # independent (prompt, files) completion, so only the aggregation below
    # needs to stay in `required` order for deterministic output.
    not_applicable = {
        mt: reason
        for mt in required
        if metadata.get(mt)
        for reason in [applicability(metadata[mt], iac, inventory)]
        if reason
    }
    referenced = {
        mt: reason
        for mt in required
        if metadata.get(mt) and mt not in not_applicable
        for reason in [externally_provided(metadata[mt], iac)]
        if reason
    }
    assessable = [
        mt
        for mt in required
        if metadata.get(mt)
        and metadata[mt].get("files_glob")
        and llm_client is not None
        and mt not in not_applicable
        and mt not in referenced
    ]
    assessed: dict[str, tuple[str, dict[str, Any] | None, str | None]] = {}
    if assessable:
        done = 0
        _tick(
            f"▶ Layer 4 (risk-management): judging {len(assessable)} mitigations "
            f"({max(1, max_workers)} workers)…"
        )
        assert llm_client is not None and prompt is not None and contract is not None
        with ThreadPoolExecutor(max_workers=max(1, max_workers)) as pool:
            futures = {}
            for i, mt in enumerate(assessable):
                if i:
                    time.sleep(_SUBMIT_STAGGER_SECONDS)
                futures[
                    pool.submit(
                        _assess_mitigation,
                        llm_client,
                        workspace,
                        inventory,
                        mt,
                        metadata[mt],
                        prompt,
                        contract,
                        max_bytes,
                        iac["files"] if iac else None,
                        iac_note,
                    )
                ] = mt
            for future in as_completed(futures):
                mt = futures[future]
                done += 1
                assessed[mt] = future.result()
                _tick(
                    f"Layer 4 (risk-management): {mt} done [{done}/{len(assessable)}]"
                )

    findings: list[Finding] = []
    coverage: list[dict[str, str]] = []
    for mitigation_type in required:
        meta = metadata.get(mitigation_type)
        if meta is None:
            coverage.append(
                {
                    "mitigation_type": mitigation_type,
                    "title": mitigation_type,
                    "status": "unknown_type",
                }
            )
            notes.append(
                f"Layer 4: DataRobot risk-management requires mitigation type "
                f"'{mitigation_type}', which isn't in this skill's assessment "
                "metadata yet (run validate_risk_management_mapping.py to confirm)."
            )
            continue

        if mitigation_type in not_applicable:
            coverage.append(
                {
                    "mitigation_type": mitigation_type,
                    "title": meta["title"],
                    "status": "not_applicable",
                    "reason": not_applicable[mitigation_type],
                }
            )
            continue

        if mitigation_type in referenced:
            coverage.append(
                {
                    "mitigation_type": mitigation_type,
                    "title": meta["title"],
                    "status": "referenced",
                    "reason": referenced[mitigation_type],
                }
            )
            continue

        verdict, item, skip_reason = "skipped", None, None
        if not meta.get("files_glob"):
            skip_reason = "organizational requirement, not assessable from code"
        elif mitigation_type in assessed:
            verdict, item, skip_reason = assessed[mitigation_type]

        if verdict == "pass":
            coverage.append(
                {
                    "mitigation_type": mitigation_type,
                    "title": meta["title"],
                    "status": "pass",
                }
            )
            continue

        if verdict == "gap":
            findings.append(
                _finding_for_mitigation(mitigation_type, meta, item, iac, offline)
            )
            coverage.append(
                {
                    "mitigation_type": mitigation_type,
                    "title": meta["title"],
                    "status": "gap",
                }
            )
            continue

        if skip_reason:
            notes.append(f"Layer 4: {mitigation_type} not assessed, {skip_reason}.")
        coverage.append(
            {
                "mitigation_type": mitigation_type,
                "title": meta["title"],
                "status": "not_assessed",
            }
        )

    if not_applicable:
        notes.append(
            f"Layer 4: {len(not_applicable)} mitigation(s) not applicable to this repo "
            f"(target type {repo_target_type(iac, inventory) or 'none'}): "
            + ", ".join(metadata[mt]["title"] for mt in not_applicable)
            + "."
        )
    if referenced:
        notes.append(
            f"Layer 4: {len(referenced)} mitigation(s) are provided by an existing "
            f"deployment referenced in {(iac or {}).get('referenced_deployment_file')}; verify "
            "them on that deployment in DataRobot."
        )
    if iac and iac.get("inactive_variants"):
        active = (
            ", ".join(f"{k} -> {v}" for k, v in sorted(iac["variants"].items()))
            or "none selected"
        )
        notes.append(
            f"Layer 4: {len(iac['inactive_variants'])} inactive configuration variant(s) "
            f"under configurations/ were not counted as deployed (active: {active})."
        )
    unassessed = sum(1 for row in coverage if row["status"] == "not_assessed")
    if findings or unassessed:
        _tick(
            f"{len(findings)} mitigation(s) required by '{policy_name}' are unsatisfied; "
            f"{unassessed} not assessed."
        )
    return findings, coverage, notes, iac or {}
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall