inehemiasm/AiAssistant / .agents/skills/developing-genkit-python
developing-genkit-python
FlaggedDevelop AI-powered applications using Genkit in Python. Use when the user asks about Genkit, AI agents, flows, or tools in Python, or when encountering Genkit errors, import issues, or API problems.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.4 KB |
| references/common-errors.md | markdown | 2.1 KB |
| references/dev-workflow.md | markdown | 2.5 KB |
| references/dotprompt.md | markdown | 2.7 KB |
| references/evals.md | markdown | 2.8 KB |
| references/examples.md | markdown | 3.8 KB |
| references/fastapi.md | markdown | 7.4 KB |
| references/setup.md | markdown | 1.2 KB |
references/evals.md
View on GitHub# Evals — Genkit Python
## Two types of evaluators
1. **Built-in** — ship with `genkit-plugin-evaluators`, register with `register_genkit_evaluators(ai)`
2. **BYO (LLM-based)** — define your own scoring logic with `ai.define_evaluator()`
## Install
```bash
uv add genkit-plugin-evaluators
```
## Dataset format
A JSON file, one object per test case:
```json
[
{"testCaseId": "case1", "input": "x", "output": "banana", "reference": "ba?a?a"},
{"testCaseId": "case2", "input": "x", "output": "apple", "reference": "ba?a?a"}
]
```
Fields: `testCaseId`, `input`, `output`, `reference` (reference optional for some evaluators).
## Built-in evaluators
```python
from genkit.plugins.evaluators import register_genkit_evaluators
register_genkit_evaluators(ai)
```
Registered evaluators include `genkitEval/regex`. Run via CLI:
```bash
genkit eval:run datasets/my_dataset.json --evaluators=genkitEval/regex
```
## BYO evaluator
```python
from genkit.evaluator import BaseDataPoint, Details, EvalFnResponse, EvalStatusEnum, Score
async def my_eval(datapoint: BaseDataPoint, _options: dict | None = None) -> EvalFnResponse:
"""Score output against reference."""
output = str(datapoint.output or '')
reference = str(datapoint.reference or '')
passed = output.strip() == reference.strip()
return EvalFnResponse(
test_case_id=datapoint.test_case_id or '',
evaluation=Score(
score=1.0 if passed else 0.0,
status=EvalStatusEnum.PASS if passed else EvalStatusEnum.FAIL,
details=Details(reasoning='Exact match check'),
),
)
ai.define_evaluator(
name='byo/my_eval',
display_name='My Eval',
definition='Checks exact match of output vs reference.',
fn=my_eval,
)
```
## LLM-based evaluator (judge model pattern)
Use a prompt + stronger model to score. See `py/samples/evaluators/src/main.py` for full examples (`byo/maliciousness`, `byo/answer_accuracy`).
Core pattern:
```python
async def llm_eval(datapoint: BaseDataPoint, _options: dict | None = None) -> EvalFnResponse:
prompt = ai.prompt('my_judge_prompt')
rendered = await prompt.render(input={'output': str(datapoint.output), 'reference': str(datapoint.reference)})
response = await ai.generate(model='googleai/gemini-flash-latest', messages=rendered.messages)
score = float(response.text.strip())
return EvalFnResponse(
test_case_id=datapoint.test_case_id or '',
evaluation=Score(score=score, status=EvalStatusEnum.PASS if score >= 0.5 else EvalStatusEnum.FAIL),
)
```
## Run evals via CLI
```bash
genkit eval:run datasets/my_dataset.json --evaluators=byo/my_eval
genkit eval:run datasets/my_dataset.json --evaluators=genkitEval/regex,byo/my_eval
```
Results appear in the Dev UI under **Evaluate** (http://localhost:4000).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |