Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/gap_analysis/models.py

View on GitHub
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Core data types shared across the engine."""

from __future__ import annotations

from dataclasses import asdict, dataclass, field, fields
from enum import Enum
from typing import Any


class Severity(str, Enum):
    CRITICAL = "critical"
    HIGH = "high"
    MEDIUM = "medium"
    LOW = "low"

    @property
    def rank(self) -> int:
        return {"critical": 0, "high": 1, "medium": 2, "low": 3}[self.value]


# Pillar id -> human label, used for grouping in the report.
PILLARS = {
    "SEC": "Security",
    "IDN": "Identity & Access",
    "AIG": "AI/LLM Governance",
    "OPS": "Operations & Observability",
    "REL": "Reliability",
    "ITA": "IT Conformance",
    "POL": "Regulatory Policy",
}


@dataclass
class Finding:
    """A single detected gap. Never carries a raw secret value."""

    condition_id: str
    pillar: str
    severity: Severity
    title: str
    file: str | None = None
    line: int | None = None
    evidence: str = ""
    explanation: str = ""
    remediation: str = ""
    fix_type: str = "advisory"  # auto | assisted | advisory
    fix_strategy: str | None = None
    fix_risk: str = "none"  # plumbing | business_logic | none (blast radius)
    confidence: str = "high"  # high | medium | low
    layer: int = 0
    detector: str = ""
    structural: bool = False  # only meaningful for findings with no taxonomy.yaml
    # Layer 2 provenance: the root cause the model grouped locations under, whether
    # a second verification pass confirmed the finding, and the remediation shape
    # that pass saw ("patch" | "structural"; empty defers to the taxonomy flag).
    root_cause: str = ""
    verified: bool = False
    verification: str = ""
    shape: str = ""
    # entry (dynamically-generated Layer 4 findings); posture.py falls back to
    # this when a taxonomy lookup by condition_id finds nothing.
    # Structured remediation (Layer 4): ordered steps, the docs page for the
    # DataRobot feature, what the fix attaches to and whether the repo has it,
    # and the fix path (pulumi | api | automatic | organizational) with the
    # resource it requires (deployment | custom_model | "").
    steps: list[str] = field(default_factory=list)
    docs_url: str = ""
    docs_topic: str = ""
    prerequisite: str = ""
    fix_via: str = ""
    fix_requires: str = ""

    def to_dict(self) -> dict[str, Any]:
        d = asdict(self)
        d["severity"] = self.severity.value
        return d

    @property
    def dedup_key(self) -> tuple[str, str | None, int | None, str]:
        """Identity for collapsing/matching findings: file-level findings (no
        line) stay distinct per evidence so N CVEs in one manifest aren't one."""
        return (
            self.condition_id,
            self.file,
            self.line,
            self.evidence if self.line is None else "",
        )

    @classmethod
    def from_dict(cls, d: dict[str, Any]) -> Finding:
        data = dict(d)
        data["severity"] = Severity(data.get("severity", "medium"))
        known = {f.name for f in fields(cls)}
        return cls(**{k: v for k, v in data.items() if k in known})


@dataclass
class ConditionSkip:
    """Recorded when a (usually relational) condition could not be evaluated."""

    condition_id: str
    reason: str


@dataclass
class AnalysisResult:
    findings: list[Finding] = field(default_factory=list)
    skipped: list[ConditionSkip] = field(default_factory=list)
    inventory: dict[str, Any] = field(default_factory=dict)
    notes: list[str] = field(default_factory=list)
    posture: dict[str, Any] = field(
        default_factory=dict
    )  # remediation posture (see posture.py)
    # Every mitigation considered for Layer 4 (DataRobot risk-management), not just
    # the ones that became findings, each is {mitigation_type, title, status},
    # status one of "gap" | "pass" | "not_applicable" | "not_assessed". Lets the
    # report render regulatory coverage without a fixed checklist to compare
    # against (Layer 4 has no static condition list, see taxonomy.yaml).
    regulatory_coverage: list[dict[str, str]] = field(default_factory=list)
    # The repo's pulumi-datarobot footprint as seen by Layer 4 (see
    # risk_management._detect_iac); empty when no Pulumi program was found.
    iac: dict[str, Any] = field(default_factory=dict)
    # LLM Gateway token usage for this run, per phase and in total (see
    # datarobot_skills_utils.opencode.UsageMeter); empty when no LLM ran.
    usage: dict[str, Any] = field(default_factory=dict)

    def to_dict(self) -> dict[str, Any]:
        return {
            "findings": [f.to_dict() for f in self.findings],
            "skipped": [asdict(s) for s in self.skipped],
            "notes": list(self.notes),
            "posture": self.posture,
            "regulatory_coverage": self.regulatory_coverage,
            "iac": self.iac,
            "usage": self.usage,
            "inventory": {k: v for k, v in self.inventory.items() if k != "files"},
        }

    @classmethod
    def from_dict(cls, d: dict[str, Any]) -> AnalysisResult:
        result = cls()
        result.findings = [Finding.from_dict(f) for f in d.get("findings", [])]
        result.skipped = [ConditionSkip(**s) for s in d.get("skipped", [])]
        result.notes = list(d.get("notes", []))
        result.posture = d.get("posture") or {}
        result.regulatory_coverage = d.get("regulatory_coverage") or []
        result.iac = d.get("iac") or {}
        result.usage = d.get("usage") or {}
        result.inventory = d.get("inventory") or {}
        return result

    def by_severity(self) -> list[Finding]:
        return sorted(
            self.findings,
            key=lambda f: (f.severity.rank, f.pillar, f.condition_id),
        )

    def counts(self) -> dict[str, int]:
        out = {s.value: 0 for s in Severity}
        for f in self.findings:
            out[f.severity.value] += 1
        return out
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall