Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

references/pillars-and-layers.md

View on GitHub
# The Assessment Framework

The engine evaluates every submitted repository against a registry of 41 static
conditions, defined in `scripts/taxonomy.yaml`, plus a dynamic regulatory layer
whose checks come from the org's own DataRobot risk-management policy at run
time (see below). Each static condition belongs to exactly one pillar and is
evaluated by exactly one layer.

## Seven risk pillars

| Pillar | ID prefix | # conditions | What it evaluates |
|---|---|---|---|
| Security | `SEC` | 13 | Secret exposure, prompt-injection vectors, encryption, input validation |
| Identity | `IDN` | 4 | Human/shared identity, credential rotation, RBAC, over-permissioning |
| AI Governance | `AIG` | 8 | Guardrails, model pinning, approved models, evals, human-in-loop, cost controls, grounding, prompt versioning |
| Reliability | `REL` | 6 | Tests, CI, lint gates, lockfiles, retry logic, resilience, fallback paths |
| Ops | `OPS` | 3 | Structured logging, tracing, health checks |
| IT Conformance | `ITA` | 7 | Python version floor, library allow/deny, approved models, licenses, base images |
| Regulatory & Policy | `POL` | dynamic | Whatever mitigations the org's DataRobot risk-management policy requires (EU AI Act by default). Not defined in `taxonomy.yaml`; finding ids look like `POL-DR-<MITIGATION-TYPE>`. |

## Four evaluation layers

| Layer | Mechanism | # conditions | Degrades to |
|---|---|---|---|
| 1 - Deterministic | A built-in secret scanner (its findings get the same second-look verification pass as Layer 2 when an LLM is available, so a resource URN or type name that merely contains "credential" is dropped, and an unconfirmed finding is never auto-fixed), `trivy` (dependency CVEs, secrets, IaC misconfiguration, dependency licenses) with `pip-audit` and `npm audit` as the fallback, `gitleaks` for secrets in git history, `hadolint` for Dockerfiles, `semgrep`, plus presence checks: tests, CI, lint/type-check gate, lockfiles, dependency updates, vulnerability scanning, CODEOWNERS, action pinning | 15 | Optional scanners that aren't installed are skipped and named in the report. Layer 1 always runs. |
| 2 - LLM reasoning | A per-condition prompt template (under `scripts/prompts/`) reads the condition's `files_glob` and reasons about the code. Files are shown with numbered lines and each finding's quoted evidence is snapped onto the line that holds it. A file over 48 KB goes in as an excerpt (header, definition lines, and windows around its `hint_patterns` hits, with omitted ranges marked and real line numbers) instead of a head-truncated blob. Checks that time out are listed in the coverage block with the `--select ... --llm-timeout` command that reruns just those. Per-condition `hint_patterns` are grepped across the whole repo and shown to the model as evidence outside its file cap. Every finding then gets a second verification call that re-reads the cited region and refutes, weakens or confirms it (`--no-verify` disables it); refuted findings are dropped and named in Engine Notes, and only verified findings may be auto-fixed. Locations sharing a `root_cause` merge into one finding. Relational checks (`SEC-001`, `IDN-003`) require *both* file groups involved, or are marked skipped-with-reason, never guessed. When the Pulumi program deploys the repo as a DataRobot custom application, every prompt is told that identity headers (`x-user-id`, `x-datarobot-api-key`, ...) are set by the platform proxy, so reading them is not flagged as trusting client input. | 20 | Skipped, with a stated reason, if no LLM client is configured. |
| 3 - Conformance | Compares the repo's declared Python version, dependencies/imports, model ids, licenses, and base images against the merged policy. | 6 | Runs fully offline; no LLM needed. |
| 4 - Regulatory | Fetches the org's DataRobot risk-management policy (named by `regulatory.policy_name`, default "EU AI Act"); for each required mitigation, the LLM judges whether the repo shows evidence it is implemented, or configured to be provided by DataRobot at deployment time. Unsatisfied mitigations become findings. Each mitigation carries the platform's applicability rules (`applicable_entity`, `applicable_target_types` in `risk_management_mitigations.yaml`); checks outside the target type the repo would deploy as (AgenticWorkflow, TextGeneration, or a predictive type) are reported as "not applicable", as are all entity-scoped checks for a repo with no model or LLM path. A missing deployment or a direct LLM Gateway call never exempts a check. Only the active Pulumi configuration counts (a `configurations/` variant is deployed only when the symlink selects it); when an inactive variant already declares a Deployment, the remedy is to select it, otherwise an agentic deployment. A program that only references a deployment another stack owns (`Deployment.get`) reports the affected checks as "provided by an existing deployment, verify there". Needs `DATAROBOT_API_TOKEN`/`DATAROBOT_ENDPOINT`. | dynamic | Skipped, with a stated reason, if DataRobot risk-management isn't reachable (no local fallback checklist). With `--no-llm`, required mitigations are reported as "required, not assessed" instead of judged. |

## Severity scale

Every finding carries one of: `critical`, `high`, `medium`, `low`. For Layers 1-3
severities are condition defaults in `taxonomy.yaml`; a policy file can override
them per-condition via `severity_overrides`. Layer 4 findings take their default
severity from `scripts/risk_management_mitigations.yaml` instead.

## Fix classification

Every condition also carries a `fix_type`, which drives what the remediation step
offers:

- **auto**: a deterministic codemod (secret → env var + `.gitignore` entry,
  dependency bump, Python version pin, CI/test scaffold).
- **assisted**: the report carries a self-contained prompt for the developer's own coding agent (citation, evidence, verification note, fix guidance, safety rails); the engine never applies these itself, because an agent with the repo open can read call sites and run tests where a one-shot edit cannot
- **advisory**: written guidance only; no automated fix exists (these are also the
  findings most likely to be flagged `structural`, see
  [remediation-paths.md](remediation-paths.md)). Layer 4 findings are advisory by
  default: satisfying a risk-management mitigation almost always means adopting the
  corresponding DataRobot platform feature (deployment monitoring, GenAI Guards,
  RBAC, Model Registry documentation), not patching the repo in place. The
  exception: on a repo that already carries a pulumi-datarobot program, the
  mitigations enabled by IaC (deployment drift/accuracy/fairness settings,
  notification policies, guard configurations) become **assisted** fixes, offered as agent prompts, that
  propose the missing settings block into the existing Pulumi resources, and stop
  counting as structural.

## Extending the framework

To add a Layer-2 (LLM) check, append a condition to `scripts/taxonomy.yaml` and add
its prompt file under `scripts/prompts/`; no code change is needed. Layer-1 and
Layer-3 conditions also need a detector in `scripts/gap_analysis/scanners.py` or
`conformance.py`. This is also the mechanism for an org to layer its own
standards on top of the baseline: point `--policy` at a file that adds
`severity_overrides` or tightens `it_admin` values, and add any org-specific
conditions directly to a copy of `taxonomy.yaml` referenced via `GAP_DATA_DIR`.

Layer 4 is extended in DataRobot, not here: an admin edits the org's
risk-management policy (or authors a new one and points `regulatory.policy_name`
at it), and the next run picks it up automatically. The only skill-side asset is
`scripts/risk_management_mitigations.yaml`, remediation metadata per mitigation
*type*, which only needs a new entry when DataRobot itself introduces a new
mitigation type (run `scripts/validate_risk_management_mapping.py` to detect
that).
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 196kUpdated Claude CodeCodexVerifiedInstall