Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-gap-analysis

datarobot-gap-analysis

Flagged

Use when the user wants to assess whether an AI agent repository (DataRobot-built or not) is enterprise-ready, run a gap analysis / readiness scan against security, governance, reliability, or regulatory standards, score an agent against EU AI Act requirements, or find and fix gaps (secrets, unpinned models, missing CI, missing guardrails, over-permissioned identity, etc.) before deploying an agent to production. Works on any repository, not only ones built with DataRobot.

FileKindSize
SKILL.mdskill md17.2 KB
references/pillars-and-layers.mdmarkdown7.8 KB
references/policy-authoring.mdmarkdown6.3 KB
references/remediation-paths.mdmarkdown3.0 KB
scripts/gap_analysis/__init__.pyscript534 B
scripts/gap_analysis/_bootstrap.pyscript2.7 KB
scripts/gap_analysis/agent_prompt.pyscript2.5 KB
scripts/gap_analysis/cli.pyscript19.9 KB
scripts/gap_analysis/conformance.pyscript9.6 KB
scripts/gap_analysis/detect.pyscript24.4 KB
scripts/gap_analysis/docs.pyscript2.9 KB
scripts/gap_analysis/engine.pyscript9.3 KB
scripts/gap_analysis/ingest.pyscript3.4 KB
scripts/gap_analysis/inventory.pyscript28.0 KB
scripts/gap_analysis/llm.pyscript2.6 KB
scripts/gap_analysis/migrate.pyscript11.7 KB
scripts/gap_analysis/models.pyscript5.9 KB
scripts/gap_analysis/opencode.pyscript2.7 KB
scripts/gap_analysis/paths.pyscript1.1 KB
scripts/gap_analysis/policy.pyscript1.3 KB
scripts/gap_analysis/posture.pyscript9.6 KB
scripts/gap_analysis/remediate.pyscript20.1 KB
scripts/gap_analysis/report.pyscript20.8 KB
scripts/gap_analysis/report_html.pyscript41.0 KB
scripts/gap_analysis/risk_management.pyscript36.3 KB
scripts/gap_analysis/scanners.pyscript52.0 KB
scripts/gap_analysis/settings.pyscript2.2 KB
scripts/gap_analysis/taxonomy.pyscript5.3 KB
scripts/gap_analysis/variants.pyscript1.5 KB
scripts/policy/defaults.yamlconfig4.6 KB
scripts/prompts/_contract.mdmarkdown3.0 KB
scripts/prompts/_deployment_datarobot_app.mdmarkdown1.2 KB
scripts/prompts/_fix_contract.mdmarkdown1.3 KB
scripts/prompts/_migrate_contract.mdmarkdown1.6 KB
scripts/prompts/_verify.mdmarkdown2.2 KB
scripts/prompts/aig-001-guardrails.mdmarkdown475 B
scripts/prompts/aig-002-model-pinning.mdmarkdown446 B
scripts/prompts/aig-004-evals.mdmarkdown357 B
scripts/prompts/aig-005-human-in-loop.mdmarkdown370 B
scripts/prompts/aig-006-cost-controls.mdmarkdown368 B
scripts/prompts/aig-007-grounding.mdmarkdown406 B
scripts/prompts/aig-008-prompt-versioning.mdmarkdown363 B
scripts/prompts/fix-aig-001-guardrails.mdmarkdown441 B
scripts/prompts/fix-aig-002-model-pinning.mdmarkdown527 B
scripts/prompts/fix-aig-003-approved-model.mdmarkdown351 B
scripts/prompts/fix-aig-006-cost-controls.mdmarkdown354 B
scripts/prompts/fix-idn-003-narrow-scope.mdmarkdown415 B
scripts/prompts/fix-ita-005-base-image.mdmarkdown305 B
scripts/prompts/fix-ops-001-logging.mdmarkdown298 B
scripts/prompts/fix-ops-002-tracing.mdmarkdown375 B
scripts/prompts/fix-pol-pulumi.mdmarkdown3.1 KB
scripts/prompts/fix-rel-003-resilience.mdmarkdown361 B
scripts/prompts/fix-sec-004-redact-log.mdmarkdown418 B
scripts/prompts/fix-sec-011-injection.mdmarkdown598 B
scripts/prompts/fix-sec-012-validation.mdmarkdown374 B
scripts/prompts/fix-sec-013-encryption.mdmarkdown370 B
scripts/prompts/idn-001-human-identity.mdmarkdown548 B
scripts/prompts/idn-002-rotation.mdmarkdown410 B
scripts/prompts/idn-003-over-permissioned.mdmarkdown711 B
scripts/prompts/idn-004-rbac.mdmarkdown433 B
scripts/prompts/migrate-extract-spec.mdmarkdown1.9 KB
scripts/prompts/ops-001-logging.mdmarkdown396 B
scripts/prompts/ops-002-tracing.mdmarkdown351 B
scripts/prompts/ops-003-health.mdmarkdown363 B
scripts/prompts/rel-003-resilience.mdmarkdown401 B
scripts/prompts/rel-004-fallback.mdmarkdown410 B
scripts/prompts/risk-management-mitigation.mdmarkdown2.7 KB
scripts/prompts/sec-001-secret-in-context.mdmarkdown701 B
scripts/prompts/sec-011-injection.mdmarkdown711 B
scripts/prompts/sec-012-input-validation.mdmarkdown999 B
scripts/prompts/sec-013-encryption.mdmarkdown410 B
scripts/risk_management_mitigations.yamlconfig32.0 KB
scripts/run_gap_analysis.pyexecutablescript1.5 KB
scripts/taxonomy.yamlconfig28.8 KB
scripts/validate_risk_management_mapping.pyexecutablescript2.0 KB

scripts/taxonomy.yaml

View on GitHub
---
# Enterprise-Readiness Gap Analysis — Condition Registry
#
# The single source of truth for what the agent looks for. A Layer-2 condition
# needs only an entry here plus its prompt file; Layer-1 and Layer-3 conditions
# also need their detector wired in scanners.py / conformance.py.
#
# Fields per condition:
#   id            stable identifier (PILLAR-NNN)
#   pillar        SEC | IDN | AIG | OPS | REL | ITA   (POL is Layer 4, generated at runtime)
#   layer         1 = deterministic scanner
#                 2 = LLM reasoning over code
#                 3 = conformance vs. policy YAML
#
# Layer 4 (regulatory / POL) conditions are not defined here: they are generated
# at runtime from the mitigations the org's DataRobot risk-management policy
# requires (see gap_analysis/risk_management.py and risk_management_mitigations.yaml).
#   severity      critical | high | medium | low   (default; policy can override)
#   title         short human-readable name
#   description   what the gap is / why it matters
#   files_glob    which files to feed the detector (list of globs)
#   relational    true if the check needs >1 file group or is skipped-with-warning
#   detector      L1: scanner name | L2/L4: prompts/<file> | L3: conformance rule key
#   remediation   one-line guidance shown in the report
#   fix_type      auto | assisted | advisory   (drives the remediation phase)
#   fix_strategy  template/codemod name (auto) or prompts/<fix file> (assisted) or null
#   fix_risk      OPTIONAL plumbing | business_logic | none — blast radius of the fix.
#                 Defaults from fix_type (auto->plumbing, assisted->business_logic,
#                 advisory->none); set explicitly to override (e.g. an assisted fix that
#                 only swaps a config value is plumbing, not business_logic).
#   structural    OPTIONAL true if this gap is an architectural deficiency that cannot be
#                 surgically patched. Feeds the remediation-posture (patch vs re-platform)
#                 decision. Any advisory finding at high/critical is treated as structural.

version: 1

conditions:

  # ─────────────────────────────────────────────────────────────────────
  # Pillar 1 — Security (SEC)
  # ─────────────────────────────────────────────────────────────────────
  - id: SEC-001
    pillar: SEC
    layer: 2
    severity: critical
    title: Secret reachable by the LLM context
    description: >
      A credential value (API key, token, password) flows into a prompt, system message, or tool description that reaches
      the model at inference time, exposing it to logging, prompt-injection exfiltration, and the model provider.
    files_glob: ["**/config.py", "**/auth.py", "**/agent.py", "**/prompt*.py", "**/*prompt*", "**/system_message*"]
    relational: true
    detector: prompts/sec-001-secret-in-context.md
    remediation: Inject secrets only into the API client/transport layer, never into model-visible text.
    fix_type: advisory
    fix_strategy: null

  - id: SEC-002
    pillar: SEC
    layer: 1
    severity: critical
    title: Long-lived static API key in code/config
    description: >
      Credential-shaped strings hardcoded in source or config (vendor keys such as sk-, AKIA…, xox…, ghp_, AIza…, or generic
      api_key/secret/token = "…") with no refresh/broker call.
    files_glob: ["**/*.py", "**/*.ts", "**/*.js", "**/*.json", "**/*.yaml", "**/*.yml", "**/*.env*", "**/*.toml"]
    relational: false
    detector: secret_scan
    remediation: Move the secret to an environment variable / secret manager and rotate it.
    fix_type: auto
    fix_strategy: secret_to_env_var

  - id: SEC-003
    pillar: SEC
    layer: 1
    severity: critical
    title: Secret in committed env vars / plaintext config
    description: >
      Credential patterns found specifically in checked-in configuration (.env, *.yaml, *.json, docker-compose*) that leak
      through version control history and CI logs.
    files_glob: ["**/.env*", "**/*.yaml", "**/*.yml", "**/*.json", "**/docker-compose*"]
    relational: false
    detector: secret_scan
    remediation: Remove from VCS, add to .gitignore, rotate, and load from a secret store.
    fix_type: auto
    fix_strategy: secret_to_env_var

  - id: SEC-004
    pillar: SEC
    layer: 1
    severity: high
    title: Secret logged or printed
    description: >
      A credential-shaped string passed to a logging/print/trace call (print(...), console.log(...), logger.x(...)). L1 surfaces
      the candidate; a L2 pass confirms it is truly sensitive.
    files_glob: ["**/*.py", "**/*.ts", "**/*.js"]
    relational: false
    detector: secret_scan
    remediation: Remove the secret from log statements; redact before logging.
    fix_type: assisted
    fix_strategy: prompts/fix-sec-004-redact-log.md

  - id: SEC-005
    pillar: SEC
    layer: 1
    severity: high
    title: Secret in git history
    description: >
      A credential was committed at some point and remains in the repository history even if the current tree no longer contains
      it; anyone with clone access can recover it.
    files_glob: ["**/*"]
    relational: false
    detector: gitleaks
    remediation: Rotate the credential first (deleting it does not revoke it), then purge it from history if the repository
      is or will be shared.
    fix_type: advisory
    fix_strategy: null

  - id: SEC-006
    pillar: SEC
    layer: 1
    severity: high
    title: Same secret across environments
    description: >
      Identical credential values appearing in more than one environment's config (dev.env, staging.yaml, prod.yaml) — extract
      all secrets and flag exact duplicates across environments.
    files_glob: ["**/dev*", "**/staging*", "**/prod*", "**/*.env*"]
    relational: true
    detector: secret_scan
    remediation: Use distinct per-environment credentials; never share prod secrets with lower envs.
    fix_type: advisory
    fix_strategy: null

  - id: SEC-010
    pillar: SEC
    layer: 1
    severity: high
    title: Dependency vulnerabilities / known CVEs
    description: >
      Declared dependencies have known published vulnerabilities.
    files_glob: ["**/requirements*.txt", "**/pyproject.toml", "**/poetry.lock", "**/package.json", "**/package-lock.json",
      "**/uv.lock"]
    relational: false
    detector: pip-audit, npm-audit, trivy
    remediation: Upgrade affected packages to the patched versions.
    fix_type: auto
    fix_strategy: bump_vulnerable_dependency

  - id: SEC-011
    pillar: SEC
    layer: 2
    severity: high
    title: Injection risk (prompt / command / deserialization / SSRF)
    description: >
      Untrusted input reaches a dangerous sink — shell execution, eval, unsafe deserialization (pickle/yaml.load), SQL string
      concatenation, server-side request building, or an LLM prompt without sanitization.
    files_glob: ["**/*.py", "**/*.ts", "**/*.js"]
    relational: false
    detector: prompts/sec-011-injection.md
    runtime_only: true
    remediation: Validate/parameterize inputs; use safe loaders; isolate untrusted content from instructions.
    fix_type: assisted
    fix_strategy: prompts/fix-sec-011-injection.md
    hint_patterns:
      - "subprocess|os\\.system|os\\.popen|shell=True|\\beval\\(|\\bexec\\("
      - "pickle\\.loads?|yaml\\.load\\(|marshal\\.loads?|execute\\(f|SELECT .*\\+|requests\\.\\w+\\(f"

  - id: SEC-012
    pillar: SEC
    layer: 2
    severity: medium
    title: Missing input validation on tool inputs & external data
    description: >
      Tool/function inputs and external API responses are consumed without type/range/schema validation.
    files_glob: ["**/tools.py", "**/tools/**", "**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/sec-012-input-validation.md
    runtime_only: true
    remediation: Add schema validation (e.g. pydantic) at trust boundaries.
    fix_type: assisted
    fix_strategy: prompts/fix-sec-012-validation.md
    hint_patterns:
      - "BaseModel|@validator|field_validator|model_validate|jsonschema|TypeAdapter"
      - "@tool|@mcp\\.tool|def \\w*tool|Field\\("

  - id: SEC-013
    pillar: SEC
    layer: 2
    severity: high
    title: Weak or missing encryption / TLS verification disabled
    description: >
      Plaintext transport, disabled certificate verification (verify=False), weak ciphers/hashes, or secrets/data persisted
      unencrypted.
    files_glob: ["**/*.py", "**/*.ts", "**/*.js"]
    relational: false
    detector: prompts/sec-013-encryption.md
    remediation: Enforce TLS with verification; encrypt data at rest; drop weak algorithms.
    fix_type: assisted
    fix_strategy: prompts/fix-sec-013-encryption.md
    hint_patterns:
      - "verify\\s*=\\s*False|CERT_NONE|check_hostname\\s*=\\s*False|http://"
      - "hashlib\\.(md5|sha1)|Fernet|encrypt|ssl_context|create_default_context"

  # ─────────────────────────────────────────────────────────────────────
  # Pillar 1 — Identity (IDN)
  # ─────────────────────────────────────────────────────────────────────
  - id: IDN-001
    pillar: IDN
    layer: 2
    severity: high
    title: Agent authenticates as a human / shared account
    description: >
      The agent uses a personal access token, a human user's OAuth token, or a single shared service account rather than its
      own dedicated workload/agent identity.
    files_glob: ["**/auth.py", "**/client.py", "**/agent.py", "**/main.py"]
    relational: false
    detector: prompts/idn-001-human-identity.md
    scope: repo
    runtime_only: true
    remediation: Provision a dedicated workload identity (e.g. service account / OIDC) for the agent.
    fix_type: advisory
    fix_strategy: null

  - id: IDN-002
    pillar: IDN
    layer: 2
    severity: medium
    title: No credential rotation / no token expiry
    description: >
      Credentials are static with no rotation mechanism or expiry handling.
    files_glob: ["**/auth.py", "**/client.py", "**/config.py"]
    relational: false
    detector: prompts/idn-002-rotation.md
    scope: repo
    runtime_only: true
    remediation: Use short-lived tokens with automatic refresh/rotation.
    fix_type: advisory
    fix_strategy: null
    hint_patterns:
      - 'expires|ttl|refresh_token|rotate|session_max_age'

  - id: IDN-003
    pillar: IDN
    layer: 2
    severity: high
    title: Over-permissioned identity
    description: >
      Scopes/roles/permissions granted to the agent are broader than the tools and resources it actually uses (admin, wildcard
      scopes, '*' resources).
    files_glob: ["**/permissions.yaml", "**/*iam*", "**/*policy*", "**/manifest*", "**/scopes*", "**/tools.py", "**/tools/**"]
    relational: true
    detector: prompts/idn-003-over-permissioned.md
    remediation: Reduce granted scopes to the minimum the code actually exercises.
    fix_type: assisted
    fix_strategy: prompts/fix-idn-003-narrow-scope.md

  - id: IDN-004
    pillar: IDN
    layer: 2
    severity: medium
    title: No least-privilege boundary / missing RBAC on actions
    description: >
      High-impact actions are not gated by role/permission checks; no authorization boundary between callers and privileged
      operations.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/idn-004-rbac.md
    scope: repo
    runtime_only: true
    remediation: Add authorization checks before privileged actions; separate roles.
    fix_type: advisory
    fix_strategy: null


  - id: SEC-014
    pillar: SEC
    layer: 1
    severity: low
    title: CI supply chain not hardened (unpinned actions, no dependency cooldown)
    description: >
      GitHub Actions referenced by a moving tag or branch, or package managers with no cooldown / minimum release age, let
      an upstream push change what runs in CI.
    files_glob: ["**/.github/workflows/**", "**/.github/dependabot.yml", "**/pyproject.toml", "**/package.json"]
    relational: false
    detector: action_pinning
    remediation: Pin third-party actions to a commit SHA; add a dependency cooldown (Dependabot `cooldown`, uv `exclude-newer`,
      npm `minimumReleaseAge`).
    fix_type: advisory
    fix_strategy: null

  - id: SEC-015
    pillar: SEC
    layer: 1
    severity: low
    title: No automated dependency updates
    description: >
      Neither Dependabot nor Renovate is configured, so vulnerable dependencies stay in place until someone notices.
    files_glob: ["**/.github/dependabot.yml", "**/renovate.json"]
    relational: false
    detector: dependency_updates
    remediation: Add `.github/dependabot.yml` covering every ecosystem in the repo, including `github-actions`.
    fix_type: advisory
    fix_strategy: null

  - id: SEC-016
    pillar: SEC
    layer: 1
    severity: medium
    title: No vulnerability scanning in CI
    description: >
      No pipeline step scans dependencies or container images for known CVEs (trivy, grype, pip-audit, npm audit, dependency-review,
      …).
    files_glob: ["**/.github/workflows/**", "**/.harness/**", "**/Taskfile*.y*ml", "**/Makefile"]
    relational: false
    detector: vuln_scan_presence
    remediation: Add a scheduled and per-PR vulnerability scan (trivy or pip-audit / npm audit) with a reviewed ignore policy.
    fix_type: advisory
    fix_strategy: null

  # ─────────────────────────────────────────────────────────────────────
  # Pillar 2 — AI/LLM Governance (AIG)
  # ─────────────────────────────────────────────────────────────────────
  - id: AIG-001
    pillar: AIG
    layer: 2
    severity: high
    title: No guardrails (toxicity / PII / jailbreak / output filtering)
    description: >
      Model inputs/outputs pass through with no moderation, PII redaction, jailbreak detection, or output validation.
    files_glob: ["**/agent.py", "**/*llm*", "**/*model*", "**/prompt*"]
    relational: false
    detector: prompts/aig-001-guardrails.md
    scope: repo
    runtime_only: true
    remediation: Add input/output guardrails (moderation, PII filter, jailbreak checks).
    fix_type: assisted
    fix_strategy: prompts/fix-aig-001-guardrails.md
    hint_patterns:
      - "guard|moderation|jailbreak|toxicity|pii|redact|OutputParser|output_validator"
    structural: true # guardrails belong in the agent architecture, not a point patch

  - id: AIG-002
    pillar: AIG
    layer: 2
    severity: medium
    title: Model not version-pinned (floating model id)
    description: >
      The model identifier has no version/date suffix, so behavior can drift silently when the provider updates the alias.
    files_glob: ["**/*.py", "**/*.ts", "**/*.yaml", "**/*.yml", "**/*.toml"]
    relational: false
    detector: prompts/aig-002-model-pinning.md
    remediation: Pin the model to a dated/versioned id.
    fix_type: assisted
    fix_strategy: prompts/fix-aig-002-model-pinning.md
    hint_patterns:
      - "model\\s*[:=]\\s*[\"']|model_id|model_name|MODEL\\b|gpt-|claude-|gemini-|llama"
    context: llm_gateway_catalog

  - id: AIG-003
    pillar: AIG
    layer: 3
    severity: high
    title: Unapproved LLM provider/model used
    description: >
      A model is used that is not on the organization's approved-model allowlist in the policy file.
    files_glob: ["**/*.py", "**/*.ts", "**/*.yaml", "**/*.yml", "**/*.toml"]
    relational: false
    detector: models_allowlist
    remediation: Switch to an approved model from the policy allowlist.
    fix_type: assisted
    fix_strategy: prompts/fix-aig-003-approved-model.md
    fix_risk: plumbing # swaps a model id; does not touch business logic

  - id: AIG-004
    pillar: AIG
    layer: 2
    severity: medium
    title: No evaluation / eval harness for prompts
    description: >
      No automated evaluation or regression tests covering prompt/agent behavior.
    files_glob: ["**/eval*", "**/test*", "**/*.py"]
    relational: false
    detector: prompts/aig-004-evals.md
    scope: repo
    runtime_only: true
    remediation: Add an eval/regression suite for prompts and agent trajectories.
    fix_type: advisory
    fix_strategy: null
    structural: true # an eval harness is a workstream, not a patch
    hint_patterns:
      - 'deepeval|ragas|promptfoo|langsmith.*evaluate|trajectory|golden'

  - id: AIG-005
    pillar: AIG
    layer: 2
    severity: high
    title: No human-in-the-loop for high-risk/irreversible actions
    description: >
      The agent can take irreversible or high-impact actions (delete, pay, send, deploy) without a human approval gate.
    files_glob: ["**/tools.py", "**/tools/**", "**/agent.py", "**/*.py"]
    relational: false
    detector: prompts/aig-005-human-in-loop.md
    scope: repo
    runtime_only: true
    remediation: Add an approval/confirmation gate before irreversible actions.
    fix_type: advisory
    fix_strategy: null

  - id: AIG-006
    pillar: AIG
    layer: 2
    severity: medium
    title: No cost/token controls or rate limiting on model calls
    description: >
      Model calls have no max-token caps, budget limits, or rate limiting, risking runaway cost.
    files_glob: ["**/*llm*", "**/*model*", "**/agent.py", "**/*.py"]
    relational: false
    detector: prompts/aig-006-cost-controls.md
    scope: repo
    runtime_only: true
    remediation: Set max-token limits, budgets, and rate limiting on model calls.
    fix_type: assisted
    fix_strategy: prompts/fix-aig-006-cost-controls.md
    hint_patterns:
      - 'max_tokens|max_output_tokens|max_completion_tokens|token_budget|budget'

  - id: AIG-007
    pillar: AIG
    layer: 2
    severity: medium
    title: No grounding/citation; hallucination risk on factual output
    description: >
      Factual outputs are produced without retrieval grounding or source citation.
    files_glob: ["**/agent.py", "**/*rag*", "**/*retriev*", "**/*.py"]
    relational: false
    detector: prompts/aig-007-grounding.md
    remediation: Ground factual answers in retrieved sources and cite them.
    fix_type: advisory
    fix_strategy: null

  - id: AIG-008
    pillar: AIG
    layer: 2
    severity: low
    title: Prompts not versioned/externalized
    description: >
      System/instruction prompts are hardcoded inline with no versioning or externalization, hindering review and rollback.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/aig-008-prompt-versioning.md
    scope: repo
    runtime_only: true
    remediation: Externalize prompts to versioned files/registry.
    fix_type: advisory
    fix_strategy: null

  # ─────────────────────────────────────────────────────────────────────
  # Pillar 3 — Ops & Reliability (OPS, REL)
  # ─────────────────────────────────────────────────────────────────────
  - id: OPS-001
    pillar: OPS
    layer: 2
    severity: medium
    title: No structured logging / audit trail of agent actions
    description: >
      Agent actions and tool calls are not logged in a structured, auditable form.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/ops-001-logging.md
    scope: repo
    runtime_only: true
    remediation: Add structured logging with an audit trail of tool calls/decisions.
    fix_type: assisted
    fix_strategy: prompts/fix-ops-001-logging.md
    hint_patterns:
      - 'logging\.getLogger|structlog|JSONFormatter|json_format|log_format|jsonlogger'
      - 'logger\.(info|warning|error|debug)\(.*extra='
      - 'instrument\(|OTLPLogExporter|LoggingHandler'

  - id: OPS-002
    pillar: OPS
    layer: 2
    severity: medium
    title: No tracing/telemetry (OpenTelemetry / DataRobot monitoring)
    description: >
      No distributed tracing or telemetry export for observability and monitoring.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/ops-002-tracing.md
    scope: repo
    runtime_only: true
    remediation: Instrument with OpenTelemetry / DataRobot agent monitoring.
    fix_type: assisted
    fix_strategy: prompts/fix-ops-002-tracing.md
    structural: true # tracing/telemetry is cross-cutting; favors re-platform when absent
    hint_patterns:
      - 'opentelemetry|instrument\(|start_as_current_span|tracer'

  - id: OPS-003
    pillar: OPS
    layer: 2
    severity: low
    title: No health checks / metrics endpoints
    description: >
      Service exposes no health/readiness or metrics endpoints.
    files_glob: ["**/*.py", "**/*.ts", "**/main.py", "**/app.py"]
    relational: false
    detector: prompts/ops-003-health.md
    scope: repo
    runtime_only: true
    remediation: Add health/readiness and metrics endpoints.
    fix_type: advisory
    fix_strategy: null
    hint_patterns:
      - '/health|/healthz|/livez|/readyz|/ready\b|/metrics'
      - 'health_endpoint|DataRobotASGIMiddleware|HealthCheck'

  - id: REL-001
    pillar: REL
    layer: 1
    severity: high
    title: No tests or low coverage
    description: >
      The repository has no test suite (or negligible coverage).
    files_glob: ["**/test*", "**/tests/**", "**/*_test.*", "**/*.spec.*"]
    relational: false
    detector: tests_presence
    remediation: Add a unit/integration test suite with meaningful coverage.
    fix_type: auto
    fix_strategy: scaffold_test_stub

  - id: REL-002
    pillar: REL
    layer: 1
    severity: medium
    title: No CI/CD pipeline
    description: >
      No continuous integration configuration found.
    files_glob: ["**/.github/workflows/**", "**/.gitlab-ci.yml", "**/azure-pipelines.yml", "**/Jenkinsfile"]
    relational: false
    detector: ci_presence
    remediation: Add a CI pipeline running lint, tests, and security scans.
    fix_type: auto
    fix_strategy: scaffold_ci_workflow

  - id: REL-003
    pillar: REL
    layer: 2
    severity: medium
    title: No error handling / retries / timeouts on external calls
    description: >
      External/API/model calls lack try-except, retries, timeouts, or circuit breakers.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/rel-003-resilience.md
    scope: repo
    runtime_only: true
    remediation: Add timeouts, bounded retries with backoff, and error handling on external calls.
    fix_type: assisted
    fix_strategy: prompts/fix-rel-003-resilience.md
    hint_patterns:
      - 'timeout\s*=|request_timeout|Timeout\('
      - 'tenacity|@retry|backoff|max_retries|Retry\('

  - id: REL-004
    pillar: REL
    layer: 2
    severity: low
    title: No graceful degradation / fallback path
    description: >
      No fallback behavior when a dependency or model is unavailable.
    files_glob: ["**/*.py", "**/*.ts"]
    relational: false
    detector: prompts/rel-004-fallback.md
    scope: repo
    runtime_only: true
    remediation: Add fallback/degraded-mode handling for dependency failures.
    fix_type: advisory
    fix_strategy: null
    hint_patterns:
      - "fallback|circuit|degrad|except .*(Timeout|ConnectionError)|retry"

  # ─────────────────────────────────────────────────────────────────────
  # Pillar 4 — IT Conformance (ITA) — all driven by policy YAML
  - id: REL-005
    pillar: REL
    layer: 1
    severity: medium
    title: No lint / format / type-check gate
    description: >
      No linter or type checker is configured (ruff, mypy, eslint, …) or nothing in CI, a Taskfile or a Makefile runs one,
      so style and type errors reach main.
    files_glob: ["**/pyproject.toml", "**/ruff.toml", "**/mypy.ini", "**/eslint.config.*", "**/Taskfile*.y*ml", "**/.github/workflows/**"]
    relational: false
    detector: lint_gate
    remediation: Configure ruff + mypy (or eslint) and run them from a `lint-check` task in CI on every pull request.
    fix_type: advisory
    fix_strategy: null

  - id: REL-006
    pillar: REL
    layer: 1
    severity: medium
    title: Dependencies not locked
    description: >
      A Python or JavaScript manifest has no lockfile (uv.lock, poetry.lock, package-lock.json, …) and no pinned requirements,
      so builds are not reproducible.
    files_glob: ["**/pyproject.toml", "**/requirements*.txt", "**/package.json"]
    relational: false
    detector: lockfile_presence
    remediation: Commit a lockfile next to every manifest (`uv lock`, `npm install` with `package-lock.json`) and verify it
      in CI (`uv lock --locked`, `npm ci`).
    fix_type: advisory
    fix_strategy: null

  # ─────────────────────────────────────────────────────────────────────
  - id: ITA-001
    pillar: ITA
    layer: 3
    severity: high
    title: Python version below required minimum
    description: >
      The declared/target Python version is below the policy minimum.
    files_glob: ["**/pyproject.toml", "**/setup.py", "**/setup.cfg", "**/.python-version", "**/runtime.txt", "**/Dockerfile"]
    relational: false
    detector: python_min_version
    remediation: Raise the project's Python version to the policy minimum.
    fix_type: auto
    fix_strategy: pin_python_version

  - id: ITA-002
    pillar: ITA
    layer: 3
    severity: medium
    title: Disallowed library used / required library missing
    description: >
      A dependency or import is on the deny list, not on a non-empty allow list, or a required library is missing.
    files_glob: ["**/requirements*.txt", "**/pyproject.toml", "**/package.json", "**/*.py"]
    relational: false
    detector: libraries_allow_deny
    remediation: Remove disallowed libraries; add required ones per policy.
    fix_type: advisory
    fix_strategy: null

  - id: ITA-003
    pillar: ITA
    layer: 3
    severity: high
    title: Unapproved LLM model (IT-admin view)
    description: >
      A model id is not on the IT-admin approved-model allowlist. Mirrors AIG-003 from the conformance angle.
    files_glob: ["**/*.py", "**/*.ts", "**/*.yaml", "**/*.yml", "**/*.toml"]
    relational: false
    detector: models_allowlist
    remediation: Use an approved model id from the policy.
    fix_type: assisted
    fix_strategy: prompts/fix-aig-003-approved-model.md
    fix_risk: plumbing # swaps a model id; does not touch business logic

  - id: ITA-004
    pillar: ITA
    layer: 3
    severity: medium
    title: OSS license non-compliance
    description: >
      A dependency carries a license on the policy deny list (e.g. copyleft).
    files_glob: ["**/requirements*.txt", "**/pyproject.toml", "**/package.json"]
    relational: false
    detector: license_check
    remediation: Replace dependencies with denied licenses; obtain legal approval if required.
    fix_type: advisory
    fix_strategy: null

  - id: ITA-005
    pillar: ITA
    layer: 3
    severity: medium
    title: Unapproved base image / runtime
    description: >
      A Dockerfile FROM image is not on the approved base-image allowlist.
    files_glob: ["**/Dockerfile", "**/*.dockerfile", "**/containerfile"]
    relational: false
    detector: base_image_allowlist
    remediation: Switch to an approved base image.
    fix_type: assisted
    fix_strategy: prompts/fix-ita-005-base-image.md
    fix_risk: plumbing # swaps a Dockerfile base image; does not touch business logic

  - id: ITA-006
    pillar: ITA
    layer: 1
    severity: low
    title: No CODEOWNERS
    description: >
      No CODEOWNERS file names a team accountable for reviewing changes.
    files_glob: ["**/CODEOWNERS"]
    relational: false
    detector: codeowners_presence
    remediation: Add `.github/CODEOWNERS` naming a real GitHub team for the whole repo.
    fix_type: advisory
    fix_strategy: null

  - id: ITA-007
    pillar: ITA
    layer: 1
    severity: medium
    title: Container or infrastructure misconfiguration
    description: >
      A Dockerfile, Kubernetes manifest or Terraform/Pulumi file violates container and IaC hardening rules (running as root,
      unpinned base images, secrets in build args, permissive settings).
    files_glob: ["**/Dockerfile*", "**/*.dockerfile", "**/*.yaml", "**/*.yml", "**/*.tf"]
    relational: false
    detector: trivy_misconfig, hadolint
    remediation: Fix the flagged instruction; the rule id links to the tool's documentation for the exact change.
    fix_type: advisory
    fix_strategy: null
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall