Skip to content

inehemiasm/AiAssistant / .agents/skills/developing-genkit-python

developing-genkit-python

Flagged

Develop AI-powered applications using Genkit in Python. Use when the user asks about Genkit, AI agents, flows, or tools in Python, or when encountering Genkit errors, import issues, or API problems.

FileKindSize
SKILL.mdskill md2.4 KB
references/common-errors.mdmarkdown2.1 KB
references/dev-workflow.mdmarkdown2.5 KB
references/dotprompt.mdmarkdown2.7 KB
references/evals.mdmarkdown2.8 KB
references/examples.mdmarkdown3.8 KB
references/fastapi.mdmarkdown7.4 KB
references/setup.mdmarkdown1.2 KB

references/evals.md

View on GitHub
# Evals — Genkit Python

## Two types of evaluators

1. **Built-in** — ship with `genkit-plugin-evaluators`, register with `register_genkit_evaluators(ai)`
2. **BYO (LLM-based)** — define your own scoring logic with `ai.define_evaluator()`

## Install

```bash
uv add genkit-plugin-evaluators
```

## Dataset format

A JSON file, one object per test case:
```json
[
  {"testCaseId": "case1", "input": "x", "output": "banana", "reference": "ba?a?a"},
  {"testCaseId": "case2", "input": "x", "output": "apple",  "reference": "ba?a?a"}
]
```

Fields: `testCaseId`, `input`, `output`, `reference` (reference optional for some evaluators).

## Built-in evaluators

```python
from genkit.plugins.evaluators import register_genkit_evaluators
register_genkit_evaluators(ai)
```

Registered evaluators include `genkitEval/regex`. Run via CLI:
```bash
genkit eval:run datasets/my_dataset.json --evaluators=genkitEval/regex
```

## BYO evaluator

```python
from genkit.evaluator import BaseDataPoint, Details, EvalFnResponse, EvalStatusEnum, Score

async def my_eval(datapoint: BaseDataPoint, _options: dict | None = None) -> EvalFnResponse:
    """Score output against reference."""
    output = str(datapoint.output or '')
    reference = str(datapoint.reference or '')
    passed = output.strip() == reference.strip()
    return EvalFnResponse(
        test_case_id=datapoint.test_case_id or '',
        evaluation=Score(
            score=1.0 if passed else 0.0,
            status=EvalStatusEnum.PASS if passed else EvalStatusEnum.FAIL,
            details=Details(reasoning='Exact match check'),
        ),
    )

ai.define_evaluator(
    name='byo/my_eval',
    display_name='My Eval',
    definition='Checks exact match of output vs reference.',
    fn=my_eval,
)
```

## LLM-based evaluator (judge model pattern)

Use a prompt + stronger model to score. See `py/samples/evaluators/src/main.py` for full examples (`byo/maliciousness`, `byo/answer_accuracy`).

Core pattern:
```python
async def llm_eval(datapoint: BaseDataPoint, _options: dict | None = None) -> EvalFnResponse:
    prompt = ai.prompt('my_judge_prompt')
    rendered = await prompt.render(input={'output': str(datapoint.output), 'reference': str(datapoint.reference)})
    response = await ai.generate(model='googleai/gemini-flash-latest', messages=rendered.messages)
    score = float(response.text.strip())
    return EvalFnResponse(
        test_case_id=datapoint.test_case_id or '',
        evaluation=Score(score=score, status=EvalStatusEnum.PASS if score >= 0.5 else EvalStatusEnum.FAIL),
    )
```

## Run evals via CLI

```bash
genkit eval:run datasets/my_dataset.json --evaluators=byo/my_eval
genkit eval:run datasets/my_dataset.json --evaluators=genkitEval/regex,byo/my_eval
```

Results appear in the Dev UI under **Evaluate** (http://localhost:4000).
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 466kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 319kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 319kUpdated Claude CodeCodexVerifiedInstall