Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/policy/defaults.yaml

View on GitHub
---
# Enterprise-Readiness Gap Analysis — Default Policy
#
# These are opinionated, sensible defaults. An IT admin / team supplies their
# own policy file (via --policy) which is DEEP-MERGED over these defaults:
#   - scalars are overridden
#   - lists are REPLACED (not concatenated) unless the key ends in `_add`
#   - missing keys fall back to these defaults
#
# Semantics:
#   libraries.allow  -> empty list means "allow all"; non-empty = strict allowlist
#   *.deny           -> always enforced
#   models.allow     -> glob patterns ('*' wildcard) of approved model ids

it_admin:

  python:
    # ITA-001 — minimum acceptable Python version
    min_version: "3.11"

  libraries:
    # ITA-002 — empty allow = allow-all; add entries to enforce a strict allowlist
    allow: []
    # Packages/usages that are never permitted
    deny:
      - "telnetlib" # insecure protocol
      - "ftplib" # insecure protocol
      - "pycrypto" # unmaintained, known CVEs (use 'cryptography')
    # Packages that MUST be present
    require: []

  models:
    # AIG-003 / ITA-003: approved LLM model ids (glob patterns). Models served by
    # the org's DataRobot LLM Gateway (`dr llm-gateway list`) are approved as well
    # when the `dr` CLI is available; the provider families below are the ones the
    # gateway and DataRobot application templates use.
    allow:
      - "anthropic/claude-*"
      - "datarobot/*"
      - "datarobotdev/*"
      - "azure/gpt-*"
      - "bedrock/anthropic.claude-*"
      - "bedrock/amazon.nova-*"
      - "vertex_ai/gemini-*"
      - "openai/gpt-5*"
      - "openai/gpt-4.1*"
      - "google/gemini-2.5*"

  licenses:
    # ITA-004 — denied OSS licenses (SPDX ids)
    deny:
      - "GPL-2.0"
      - "GPL-3.0"
      - "AGPL-3.0"
      - "SSPL-1.0"

  base_images:
    # ITA-005 — approved container base images (glob patterns)
    allow:
      - "python:3.11*"
      - "python:3.12*"
      - "python:3.13*"
      - "datarobot/*"
      - "datarobotdev/*"
      - "datarobot_notebook*" # local kernel image built by DataRobot notebook execution environments

regulatory:
  # Which regulatory packs (Layer 4) to run. Extensible: add gdpr, soc2, nist_ai_rmf.
  # The eu_ai_act pack means "ask DataRobot risk-management what it requires,"
  # not "run a fixed set of checks." No DataRobot connection means no findings
  # for this pack, not a fallback to a generic checklist.
  packs:
    - "eu_ai_act"
  # Name (or id) of the DataRobot risk-management policy to look up for the
  # eu_ai_act pack. Change this if the org's policy isn't named "EU AI Act"
  # (e.g. a custom policy under a different name). On a name collision the
  # org's own policy wins over the built-in; a policy id is the unambiguous
  # form when several tenant policies share a name.
  policy_name: "EU AI Act"

# Per-condition severity overrides (key = condition id)
severity_overrides: {}
# Example:
# SEC-004: critical

scan:
  # Globs excluded from all scanning
  exclude:
    - "**/.git/**"
    - "**/node_modules/**"
    - "**/.venv/**"
    - "**/venv/**"
    - "**/dist/**"
    - "**/build/**"
    - "**/__pycache__/**"
    - "**/*.min.js"
  # Cap on file size (bytes) fed to the LLM per Layer-2 check
  max_file_bytes: 200000

posture:
  # Remediation-posture decision gate (patch vs re-platform). The score is the share of
  # weighted risk that comes from STRUCTURAL gaps (architectural deficiencies that can't be
  # surgically patched). See src/gap_analysis/posture.py.
  patch_max: 0.25 # score <= this -> PATCH (patch in place)
  replatform_min: 0.50 # score >= this -> RE-PLATFORM (migrate to af-components)
  replatform_structural_count: 4 # this many high/critical structural gaps forces RE-PLATFORM
  max_drivers: 8 # how many structural drivers to list in the report

report:
  # Severities that make the CLI exit non-zero (CI gating)
  fail_on:
    - "critical"
    - "high"

remediation:
  # Fix tiers the engine applies itself. Only deterministic codemods ("auto") run;
  # "assisted" findings ship an agent prompt in the report instead.
  allow_fix_types:
    - "auto"
  # Risk classes that may be applied in a blanket "fix all" (no explicit ids). Business-
  # logic-touching fixes are intentionally excluded so they are only applied when a human
  # names the condition id — protecting business logic in someone else's agent.
  #   plumbing       — config/manifests/scaffolding only (safe to sweep)
  #   business_logic — edits application code (require explicit per-finding opt-in)
  auto_apply_risk:
    - "plumbing"
  # Fixes land on a gap-fixes/<timestamp> branch. The engine never pushes or
  # opens a pull request; that stays a separate, explicit human step.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall