Skip to content

om-network/betayum / .agents/skills/evidence-automation

evidence-automation

Verified

How the evidence automation script system works — runtime contract, secrets, tools, output format, and the chat route. Use when building or extending evidence automation features.

Evidence Automation System

Architecture

The automation system lets an LLM generate Python scripts that collect compliance evidence from external APIs. Scripts are stored as scriptDraft on EvidenceAutomation and executed in-process by the NestJS API.

User chat → chat/route.ts (Next.js) → streamText (GPT-4o)
  → storeToS3 tool → PUT /v1/tasks/:taskId/automations/:automationId/draft-script
  → runScript tool  → POST .../draft-script/run
    → AutomationScriptExecutorService → python3 script with secrets as env vars

Chat Route

File: apps/app/src/app/api/tasks-automations/chat/route.ts

The LLM is instructed to:

  • Always write a complete Python script and call storeToS3 on every response
  • Never ask clarifying questions — make reasonable assumptions
  • Use promptForSecret after writing a script if credentials are needed
  • Never call runScript automatically — only on explicit user request

Available tools for the LLM:

ToolPurpose
storeToS3Save the generated Python script as draft
runScriptExecute the current draft (pass secretRefs array)
promptForSecretAsk user for a credential — stored encrypted, injected as env var
promptForInfoAsk user for non-secret config (IDs, URLs, etc.)

allowedTools on the EvidenceAutomation record gates which optional tools are enabled. null means all tools are enabled.

Script Runtime Contract

Executor: apps/api/src/tasks/automations/automation-script-executor.service.ts

Scripts run as: python3 script.py with env vars injected.

Environment:

  • process.env from the Node.js API process (inherits all server env vars)
  • Decrypted secrets from Secret table, keyed by secret.name

Timeout: 60 seconds hard limit.

Required output: Script must print a single JSON object to stdout:

import json, os

def main():
    # ... collect evidence ...
    return {
        "success": True,       # or False on failure
        "data": { ... },       # evidence payload
        "error": None,         # string if success=False
    }

if __name__ == "__main__":
    result = main()
    print(json.dumps(result))

The executor parses stdout as JSON. If parsed.success === true, the run is marked completed. Anything on stderr is captured as logs.

Secrets Pattern

Secrets are stored encrypted in the Secret table (organizationId, name, value, category).

When a script declares secretRefs, they are resolved and injected as env vars:

// secretRefs passed at run time:
[{ name: "GCP_SERVICE_ACCOUNT_KEY", category: "automation" }]

// In Python script:
import os, json
key = json.loads(os.environ["GCP_SERVICE_ACCOUNT_KEY"])

The LLM calls promptForSecret to trigger the UI to collect and store the secret. The secretName becomes the env var name.

Run Lifecycle

status: pending → running → completed | failed

Stored in EvidenceAutomationRun:

  • version: null for draft runs, version: N for published version runs
  • output: the parsed JSON from stdout
  • error: error string on failure
  • success: boolean

Key API Endpoints

MethodPathPurpose
PUT/v1/tasks/:taskId/automations/:automationId/draft-scriptSave draft script
GET/v1/tasks/:taskId/automations/:automationId/draft-scriptGet draft script
POST/v1/tasks/:taskId/automations/:automationId/draft-script/runRun draft (no version needed)
POST/v1/tasks/:taskId/automations/:automationId/runsRun published version
GET/v1/tasks/:taskId/automations/runs/:runIdPoll run status

Extending the System

To add a new credential source (e.g. integration OAuth tokens):

  1. Extend resolveSecrets() in automation-script-executor.service.ts to also query IntegrationConnection + IntegrationCredentialVersion
  2. Inject token as a named env var (e.g. GCP_ACCESS_TOKEN)
  3. Update buildSystemPrompt() in chat/route.ts to tell the LLM which env vars are pre-populated when an integration is connected

To add a new tool to the LLM:

  1. Add to optionalTools in chat/route.ts
  2. Add the tool name to allowedTools on EvidenceAutomation to enable it per-automation
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall