Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/gap_analysis/posture.py

View on GitHub
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Remediation-posture decision gate.

Turns a flat gap list into a recommendation: PATCH, HYBRID, or RE-PLATFORM.

The premise: fix-in-place vs. rebuild-from-scratch is a false binary. The unit of
decision is the *gap*, not the agent. Plumbing gaps (secrets, pins, scaffolding) patch
safely; STRUCTURAL gaps (no observability, no guardrails, human identity, no resilience)
can't be surgically fixed — "fixing" them means restructuring someone else's business
logic. When structural gaps dominate, lifting the business logic into a fresh, conformant
af-components base (see migrate.py) is safer than in-place surgery.

`structural` is a per-condition flag in taxonomy.yaml for Layers 1-3 (any advisory
high/critical also counts); Layer 4 findings carry it directly on the Finding instead,
since they're generated dynamically from a DataRobot risk-management policy rather
than a taxonomy.yaml entry (almost always true there, since satisfying a mitigation
nearly always means adopting a DataRobot platform feature, not patching code in
place). Thresholds live under `posture:` in the policy.
"""

from __future__ import annotations

from typing import Any

from .models import AnalysisResult, Finding, Severity
from .taxonomy import Taxonomy
from .variants import _select_variant_text, shipped_deployment_variant

# Severity weights — structural density is weighted, so a couple of critical structural
# gaps outweigh a long tail of low-severity plumbing.
_WEIGHT = {
    Severity.CRITICAL: 4.0,
    Severity.HIGH: 3.0,
    Severity.MEDIUM: 2.0,
    Severity.LOW: 1.0,
}

PATCH = "PATCH"
HYBRID = "HYBRID"
REPLATFORM = "RE-PLATFORM"

_DEFAULTS = {
    # score <= patch_max         -> PATCH
    # patch_max < score < replatform_min -> HYBRID
    # score >= replatform_min    -> RE-PLATFORM
    "patch_max": 0.25,
    "replatform_min": 0.50,
    # Absolute override: this many high/critical structural gaps forces RE-PLATFORM
    # regardless of density (a small repo can be all-structural at low total weight).
    "replatform_structural_count": 4,
    "max_drivers": 8,
}


def assess_posture(
    result: AnalysisResult,
    policy: dict[str, Any] | None = None,
    taxonomy: Taxonomy | None = None,
) -> dict[str, Any]:
    """Return {recommendation, score, structural_count, total, drivers, rationale}."""
    cfg = {**_DEFAULTS, **(policy or {}).get("posture", {})}
    tax = taxonomy or Taxonomy.load()

    findings = result.findings
    total = len(findings)
    if not total:
        return {
            "recommendation": PATCH,
            "score": 0.0,
            "structural_count": 0,
            "total": 0,
            "drivers": [],
            "rationale": "No gaps detected — nothing to remediate or re-platform.",
        }

    def is_structural(f: Finding) -> bool:
        # Findings with a taxonomy.yaml entry (Layers 1-3) defer to the
        # condition's own flag. Layer 4 findings are dynamically generated (no
        # taxonomy entry, driven by the org's live DataRobot risk-management
        # policy (see risk_management.py) and carry the flag on the Finding
        # itself instead.
        # A verification pass that saw the actual remediation shape wins over
        # the condition template.
        if f.shape in ("patch", "structural"):
            return f.shape == "structural"
        c = tax.get(f.condition_id)
        if c is not None:
            return bool(c.structural)
        return bool(f.structural)

    total_weight = sum(_WEIGHT[f.severity] for f in findings)
    structural = [f for f in findings if is_structural(f)]
    structural_weight = sum(_WEIGHT[f.severity] for f in structural)
    score = round(structural_weight / total_weight, 3) if total_weight else 0.0

    high_structural = [
        f for f in structural if f.severity in (Severity.CRITICAL, Severity.HIGH)
    ]

    # Decide.
    if (
        len(high_structural) >= cfg["replatform_structural_count"]
        or score >= cfg["replatform_min"]
    ):
        rec = REPLATFORM
    elif score <= cfg["patch_max"]:
        rec = PATCH
    else:
        rec = HYBRID

    drivers = _drivers(structural, int(cfg["max_drivers"]))
    advice = migration_advice(result.inventory, result.iac)
    return {
        "recommendation": rec,
        "score": score,
        "structural_count": len(structural),
        "total": total,
        "drivers": drivers,
        "advice": advice,
        "rationale": _rationale(
            rec, score, len(structural), total, high_structural, advice
        ),
    }


def migration_advice(
    inventory: dict[str, Any] | None, iac: dict[str, Any] | None = None
) -> str:
    """What "re-platform" means for this repo.

    A repo already generated from af-components keeps its application; the
    structural fix is to put the model or LLM path behind a DataRobot
    deployment of the right target type so guards and monitoring can attach.
    When the Pulumi program already ships a variant that declares one, the fix
    is to select it. The agent framework is never changed unless the user asks:
    frameworks without a native DataRobot template deploy through the generic
    Base flavor.
    """
    inv = inventory or {}
    iac = iac or {}
    sources = [s for s in inv.get("template_sources", []) if "af-component" in s]
    frameworks = inv.get("agent_frameworks") or []
    llm = inv.get("llm_usage") or {}
    target = iac.get("target_type") or inv.get("deploy_target")
    variant = shipped_deployment_variant(iac)
    if variant:
        text = (
            "The active Pulumi configuration deploys no datarobot.Deployment, but the "
            "repo already ships one. " + _select_variant_text(iac, variant)
        )
    elif sources:
        text = (
            f"This repo already builds on af-components ({', '.join(sources)}). Keep the "
            f"application and put the {_path_name(target)} behind {_deployment_shape(target)}, "
            "so guards, tracing and monitoring attach."
        )
        if llm.get("gateway"):
            text += (
                " The LLM calls can keep going through the DataRobot LLM Gateway from "
                "inside that model; the gateway is the provider, the deployment is the "
                "governed entity."
            )
    else:
        text = (
            "Re-platform onto af-components with the datarobot-agent-assist skill: extract "
            "the business logic into a fresh af-components base rather than restructuring "
            "in place."
        )
    if frameworks:
        native = [f["name"] for f in frameworks if f["native"]]
        other = [f["name"] for f in frameworks if not f["native"]]
        if other:
            choices = inv.get("agent_template_choices") or {}
            natives = [label for label, value in choices.items() if value != "base"]
            offered = ", ".join(natives) if natives else "a fixed set of frameworks"
            text += (
                f" The agent uses {', '.join(other)}; the DataRobot agent template offers "
                f"{offered} natively, so deploy it through its generic Base flavor, which "
                "wraps any Python agent, rather than rewriting it onto another framework."
            )
        elif native:
            text += f" The agent uses {', '.join(native)}, which has a native DataRobot agent template."
    return text


def _path_name(target: str | None) -> str:
    if target == "TextGeneration":
        return "LLM path"
    if target and target not in ("AgenticWorkflow",):
        return "model"
    return "agent or LLM path"


def _deployment_shape(target: str | None) -> str:
    if target == "TextGeneration":
        return (
            "an LLM deployment: an LLM blueprint or a datarobot.CustomModel with target "
            "type TextGeneration behind a datarobot.Deployment"
        )
    if target and target not in ("AgenticWorkflow",):
        return "a registered model behind a datarobot.Deployment"
    return (
        "an agentic deployment: a datarobot.CustomModel with target type "
        "AgenticWorkflow behind a datarobot.Deployment"
    )


def _drivers(structural: list[Finding], limit: int) -> list[dict[str, str]]:
    """One row per structural condition (deduped), worst severity first."""
    seen: dict[str, dict[str, str]] = {}
    for f in structural:
        cur = seen.get(f.condition_id)
        if cur is None or f.severity.rank < Severity(cur["severity"]).rank:
            seen[f.condition_id] = {
                "condition_id": f.condition_id,
                "severity": f.severity.value,
                "title": f.title,
            }
    ordered = sorted(seen.values(), key=lambda d: Severity(d["severity"]).rank)
    return ordered[:limit]


def _rationale(
    rec: str,
    score: float,
    structural_count: int,
    total: int,
    high_structural: list[Finding],
    advice: str = "",
) -> str:
    pct = round(score * 100)
    if rec == PATCH:
        return (
            f"Only {structural_count} of {total} gaps are structural "
            f"({pct}% of weighted risk). Patch in place — the fixes are surgical and "
            f"low-risk to existing business logic."
        )
    if rec == REPLATFORM:
        return (
            f"{len(high_structural)} high/critical structural gaps and {pct}% of "
            f"weighted risk is architectural: these gaps are closed by changing how the "
            f"system is deployed and governed, not by editing individual files. {advice}"
        )
    return (
        f"Mixed profile — {structural_count} of {total} gaps are structural "
        f"({pct}% of weighted risk). Patch the plumbing now, and plan the structural "
        f"core separately. {advice}"
    )
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall