Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/gap_analysis/migrate.py

View on GitHub
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Guided migration — extract business logic, review as a spec, scaffold a fresh base.

The logic-preserving alternative to both blind rebuild and risky in-place restructuring.
When the remediation posture is RE-PLATFORM (see posture.py), patching structural gaps
means restructuring code you don't own. Instead we:

  1. extract_spec()        — lift the business logic (prompts, tools, graph, deps) into a
                             structured spec via an LLM pass over the repo's key files;
  2. render_spec_markdown()— render it as an agent_spec.md, the HUMAN REVIEW checkpoint;
  3. scaffold_from_spec()  — assemble a migration bundle (spec + carried-over logic +
                             externalized prompts) ready for the DataRobot agent template
                             (the `datarobot-agent-assist` skill / `dr` scaffolder) to turn
                             into a runnable af-components agent.

The checkpoint is the SPEC, not a diff of unfamiliar code — that is what makes this safe.
"""

from __future__ import annotations

import fnmatch
import shutil
from pathlib import Path
from typing import Any

import yaml

from . import paths
from .llm import LLMClient, parse_json

# Files most likely to hold business logic (entrypoints, graph, tools, prompts).
# Deliberately broad on the agent/tool side, then capped by byte budget.
_LOGIC_GLOBS = [
    "**/agent*.py",
    "**/*agent*.py",
    "**/graph*.py",
    "**/*graph*.py",
    "**/main.py",
    "**/app.py",
    "**/workflow*.py",
    "**/chain*.py",
    "**/tools.py",
    "**/tools/**",
    "**/*tool*.py",
    "**/prompt*",
    "**/*prompt*",
    "**/system_message*",
    "**/instructions*",
]


def _matches(rel: str, glob: str) -> bool:
    """Glob match on the full path OR the basename, so '**/agent*.py' also hits a
    top-level 'agent.py' (fnmatch does not special-case '**')."""
    base = glob.removeprefix("**/")
    return fnmatch.fnmatch(rel, glob) or fnmatch.fnmatch(Path(rel).name, base)


def _gather_logic_files(
    workspace: Path, inventory: dict[str, Any], max_bytes: int
) -> list[tuple[str, str]]:
    """Return (rel_path, content) for candidate business-logic files, byte-capped."""
    rels: list[str] = []
    seen: set[str] = set()
    for rel in inventory.get("files", []):
        if rel not in seen and any(_matches(rel, g) for g in _LOGIC_GLOBS):
            seen.add(rel)
            rels.append(rel)
    # Stable order: agent/graph first, then tools, then prompts — most informative first.
    rels.sort(key=lambda r: ("prompt" in r.lower(), "tool" in r.lower(), r))

    out: list[tuple[str, str]] = []
    budget = max_bytes
    for rel in rels:
        p = workspace / rel
        if not p.is_file():
            continue
        try:
            text = p.read_text(errors="ignore")
        except OSError:
            continue
        snippet = text[:budget]
        out.append((rel, snippet))
        budget -= len(snippet)
        if budget <= 0:
            break
    return out


def extract_spec(
    workspace: str | Path,
    inventory: dict[str, Any],
    client: LLMClient | None,
    max_bytes: int = 120_000,
) -> dict[str, Any]:
    """Run the extraction prompt over the repo's key files; return the spec dict.

    Raises RuntimeError if no LLM client is available (extraction is inherently a
    reasoning task — there is no deterministic fallback).
    """
    if client is None:
        raise RuntimeError("migration extraction needs an LLM client (none configured)")
    workspace = Path(workspace)
    files = _gather_logic_files(workspace, inventory, max_bytes)
    if not files:
        return {
            "name": "",
            "description": "",
            "framework": "unknown",
            "model": "",
            "system_prompt": "",
            "tools": [],
            "workflow": "",
            "domain_dependencies": [],
            "carryover_files": [],
            "manual_wiring": [
                "No agent/tool/prompt files found — this may not be an agent repo."
            ],
            "confidence": "low",
        }

    prompt = paths.resolve("prompts/migrate-extract-spec.md").read_text()
    contract = (paths.prompts_dir() / "_migrate_contract.md").read_text()
    system = f"{prompt}\n\n---\n# Output contract\n{contract}"
    parts = [f"=== FILE: {rel} ===\n{content}" for rel, content in files]
    declared = inventory.get("dependencies", [])
    user = (
        "Repository key files follow. Declared dependencies: "
        f"{', '.join(declared) or 'n/a'}.\n\n" + "\n\n".join(parts)
    )

    spec = parse_json(client.complete(system, user))
    # Normalize the shape so downstream rendering/scaffolding is total.
    spec.setdefault("name", inventory.get("root", "agent").split("/")[-1])
    defaults: list[tuple[str, Any]] = [
        ("tools", []),
        ("domain_dependencies", []),
        ("carryover_files", []),
        ("manual_wiring", []),
    ]
    for k, default in defaults:
        spec.setdefault(k, default)
    for k in (
        "description",
        "framework",
        "model",
        "system_prompt",
        "workflow",
        "confidence",
    ):
        spec.setdefault(k, "")
    return spec


def render_spec_markdown(spec: dict[str, Any]) -> str:
    """Render the extracted spec as an agent_spec.md for human review/edit."""
    lines = [
        "# Migrated Agent — agent_spec.md",
        "#",
        "# Generated by the gap-analysis guided migration. REVIEW AND EDIT THIS before",
        "# scaffolding. The af-component stack (base + llm + datarobot-mcp + agent@langgraph)",
        "# provides the runtime; the fields below are your carried-over business logic.",
        "",
        f"name: {spec.get('name') or 'migrated-agent'}",
        f"description: {_yaml_scalar(spec.get('description'))}",
        f"source_framework: {spec.get('framework') or 'unknown'}",
        f"confidence: {spec.get('confidence') or 'unknown'}",
        "",
        f"model: {_yaml_scalar(spec.get('model') or 'datarobot/anthropic/claude-sonnet-4-6')}",
        "",
        "system_prompt: |",
    ]
    for ln in (spec.get("system_prompt") or "(none extracted)").splitlines() or [""]:
        lines.append(f"  {ln}")

    lines += ["", "workflow: |"]
    for ln in (spec.get("workflow") or "(none extracted)").splitlines() or [""]:
        lines.append(f"  {ln}")

    lines += ["", "tools:"]
    if spec.get("tools"):
        lines.append(yaml.safe_dump(spec["tools"], sort_keys=False).rstrip())
    else:
        lines.append("  []")

    lines += ["", "domain_dependencies:"]
    for d in spec.get("domain_dependencies") or []:
        lines.append(f"  - {d}")

    lines += ["", "carryover_files:"]
    for f in spec.get("carryover_files") or []:
        lines.append(f"  - {f}")

    if spec.get("manual_wiring"):
        lines += ["", "manual_wiring:  # reconnect these after scaffolding"]
        for m in spec["manual_wiring"]:
            lines.append(f"  - {_yaml_scalar(m)}")

    return "\n".join(lines) + "\n"


def _yaml_scalar(value: Any) -> str:
    """Quote a scalar if it could confuse the YAML reader."""
    s: str = "" if value is None else str(value)
    if s == "" or any(ch in s for ch in ":#\n") or s.strip() != s:
        return '"' + s.replace('"', '\\"').replace("\n", " ") + '"'
    return s


def scaffold_from_spec(
    spec: dict[str, Any], workspace: str | Path, dest: str | Path
) -> dict[str, Any]:
    """Assemble a migration bundle at `dest` ready for the af-components scaffolder.

    Produces, deterministically:
      dest/agent_spec.md          — the rendered, reviewable spec (handoff to the template)
      dest/prompts/system.md      — the extracted system prompt, externalized (also fixes AIG-008)
      dest/carryover/<files>      — business-logic files copied verbatim for re-insertion
      dest/README.md              — how to finish: run the datarobot-agent-assist scaffolder

    The heavy af-components template generation is intentionally DELEGATED to the
    `datarobot-agent-assist` skill / `dr` template (which the repo already depends on);
    re-implementing it here would duplicate that template.
    """
    workspace = Path(workspace)
    dest = Path(dest)
    dest.mkdir(parents=True, exist_ok=True)

    # 1. The spec — the primary artifact the scaffolder consumes.
    spec_md = render_spec_markdown(spec)
    (dest / "agent_spec.md").write_text(spec_md)

    # 2. Externalize the system prompt as a versioned file.
    prompts_dir = dest / "prompts"
    prompts_dir.mkdir(exist_ok=True)
    if spec.get("system_prompt"):
        (prompts_dir / "system.md").write_text(spec["system_prompt"].rstrip() + "\n")

    # 3. Copy carried-over business-logic files (skip anything missing or outside the repo).
    carried, missing = [], []
    carry_root = dest / "carryover"
    for rel in spec.get("carryover_files") or []:
        src = (workspace / rel).resolve()
        try:
            src.relative_to(workspace.resolve())
        except ValueError:
            missing.append(rel)  # path escapes the workspace — refuse
            continue
        if not src.is_file():
            missing.append(rel)
            continue
        out_path = carry_root / rel
        out_path.parent.mkdir(parents=True, exist_ok=True)
        shutil.copy2(src, out_path)
        carried.append(rel)

    # 4. README handoff.
    (dest / "README.md").write_text(_readme(spec, carried, missing))

    return {
        "dest": str(dest),
        "spec_path": str(dest / "agent_spec.md"),
        "carried_over": carried,
        "missing_files": missing,
        "manual_wiring": spec.get("manual_wiring") or [],
    }


def _readme(spec: dict[str, Any], carried: list[str], missing: list[str]) -> str:
    tools = (
        ", ".join(t.get("name", "?") for t in spec.get("tools") or [])
        or "none detected"
    )
    lines = [
        f"# Migration bundle — {spec.get('name') or 'migrated-agent'}",
        "",
        "This bundle was produced by the gap-analysis guided migration. It contains the",
        "**business logic** lifted from the source agent, ready to be scaffolded onto the",
        "DataRobot af-component stack — instead of restructuring the original in place.",
        "",
        "## Contents",
        "- `agent_spec.md` — the reviewable spec (model, system prompt, tools, workflow).",
        "- `prompts/system.md` — the externalized system prompt.",
        f"- `carryover/` — {len(carried)} business-logic file(s) copied verbatim.",
        "",
        (
            f"**Source framework:** {spec.get('framework') or 'unknown'} &nbsp;|&nbsp; "
            f"**Tools:** {tools} &nbsp;|&nbsp; **Confidence:** {spec.get('confidence') or 'unknown'}"
        ),
        "",
        "## Next step — scaffold the af-components agent",
        "1. **Review and edit `agent_spec.md`.** This is the human checkpoint — confirm the",
        "   prompt, tools, and workflow are faithful before generating any code.",
        "2. Run the DataRobot agent template scaffolder against it (the `datarobot-agent-assist`",
        "   skill, or `dr` with the agent template), which generates the af-component-base /",
        "   -llm / -datarobot-mcp / agent@langgraph runtime around this spec.",
        "3. Re-insert the carried-over logic from `carryover/` into the generated tools/graph.",
        "4. Re-run `run_gap_analysis` on the scaffolded agent to confirm the structural gaps",
        "   are closed.",
    ]
    if missing:
        lines += [
            "",
            "## Files referenced but not copied",
            *[f"- {m}" for m in missing],
        ]
    if spec.get("manual_wiring"):
        lines += ["", "## Manual wiring", *[f"- {m}" for m in spec["manual_wiring"]]]
    return "\n".join(lines) + "\n"
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall