om-network/betayum / .agents/skills/evidence-automation
evidence-automation
VerifiedHow the evidence automation script system works — runtime contract, secrets, tools, output format, and the chat route. Use when building or extending evidence automation features.
Evidence Automation System
Architecture
The automation system lets an LLM generate Python scripts that collect compliance evidence from external APIs. Scripts are stored as scriptDraft on EvidenceAutomation and executed in-process by the NestJS API.
User chat → chat/route.ts (Next.js) → streamText (GPT-4o)
→ storeToS3 tool → PUT /v1/tasks/:taskId/automations/:automationId/draft-script
→ runScript tool → POST .../draft-script/run
→ AutomationScriptExecutorService → python3 script with secrets as env vars
Chat Route
File: apps/app/src/app/api/tasks-automations/chat/route.ts
The LLM is instructed to:
- Always write a complete Python script and call
storeToS3on every response - Never ask clarifying questions — make reasonable assumptions
- Use
promptForSecretafter writing a script if credentials are needed - Never call
runScriptautomatically — only on explicit user request
Available tools for the LLM:
| Tool | Purpose |
|---|---|
storeToS3 | Save the generated Python script as draft |
runScript | Execute the current draft (pass secretRefs array) |
promptForSecret | Ask user for a credential — stored encrypted, injected as env var |
promptForInfo | Ask user for non-secret config (IDs, URLs, etc.) |
allowedTools on the EvidenceAutomation record gates which optional tools are enabled. null means all tools are enabled.
Script Runtime Contract
Executor: apps/api/src/tasks/automations/automation-script-executor.service.ts
Scripts run as: python3 script.py with env vars injected.
Environment:
process.envfrom the Node.js API process (inherits all server env vars)- Decrypted secrets from
Secrettable, keyed bysecret.name
Timeout: 60 seconds hard limit.
Required output: Script must print a single JSON object to stdout:
import json, os
def main():
# ... collect evidence ...
return {
"success": True, # or False on failure
"data": { ... }, # evidence payload
"error": None, # string if success=False
}
if __name__ == "__main__":
result = main()
print(json.dumps(result))
The executor parses stdout as JSON. If parsed.success === true, the run is marked completed. Anything on stderr is captured as logs.
Secrets Pattern
Secrets are stored encrypted in the Secret table (organizationId, name, value, category).
When a script declares secretRefs, they are resolved and injected as env vars:
// secretRefs passed at run time:
[{ name: "GCP_SERVICE_ACCOUNT_KEY", category: "automation" }]
// In Python script:
import os, json
key = json.loads(os.environ["GCP_SERVICE_ACCOUNT_KEY"])
The LLM calls promptForSecret to trigger the UI to collect and store the secret. The secretName becomes the env var name.
Run Lifecycle
status: pending → running → completed | failed
Stored in EvidenceAutomationRun:
version: nullfor draft runs,version: Nfor published version runsoutput: the parsed JSON from stdouterror: error string on failuresuccess: boolean
Key API Endpoints
| Method | Path | Purpose |
|---|---|---|
PUT | /v1/tasks/:taskId/automations/:automationId/draft-script | Save draft script |
GET | /v1/tasks/:taskId/automations/:automationId/draft-script | Get draft script |
POST | /v1/tasks/:taskId/automations/:automationId/draft-script/run | Run draft (no version needed) |
POST | /v1/tasks/:taskId/automations/:automationId/runs | Run published version |
GET | /v1/tasks/:taskId/automations/runs/:runId | Poll run status |
Extending the System
To add a new credential source (e.g. integration OAuth tokens):
- Extend
resolveSecrets()inautomation-script-executor.service.tsto also queryIntegrationConnection+IntegrationCredentialVersion - Inject token as a named env var (e.g.
GCP_ACCESS_TOKEN) - Update
buildSystemPrompt()inchat/route.tsto tell the LLM which env vars are pre-populated when an integration is connected
To add a new tool to the LLM:
- Add to
optionalToolsinchat/route.ts - Add the tool name to
allowedToolsonEvidenceAutomationto enable it per-automation
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |