jscraik/Agent-Skills / Plugins/aidevcon/skills/talk-obstbaum-willoughby-vibes-to-metrics
talk-obstbaum-willoughby-vibes-to-metrics
VerifiedUse when the user asks about Simon Obstbaum and Rob Willoughby's AI Native DevCon talk on measuring AI agents, output evals versus trajectory evals, instrumentation, compliance, and skill activation metrics.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.4 KB |
| agents/openai.yaml | config | 152 B |
| outline.md | markdown | 1.4 KB |
| quote.md | markdown | 2.0 KB |
| transcript.md | markdown | 47.6 KB |
quote.md
View on GitHub# Quotes -- From Vibes to Metrics: How to Actually Measure What Your AI Agents Do
Short excerpts selected from the transcript for grounding answers. Preserve transcript artifacts when quoting.
## Output evals
> They're going to be talking about from vibes to metrics and how to actually measure what your agents do. Over to you. >> Cool. So, what we're here to talk to you
Source: `L0054-L0058`
## Trajectory evals
> the levels and we correlate the levels with the output measurement that we have uh shown in in the beginning. So just when we look at uh okay why do we trust the four levels? So in terms of
Source: `L0302-L0306`
## Agent instrumentation
> different views on how to be assessing those metrics. one kind of top down looking at a correlational studies across kind of a whole bunch a whole big part of the industry, one bottoms up
Source: `L0078-L0082`
## Skill activation
> lot of time in thinking how could we even measure output and and subsequently productivity So what we found uh to work is um that we have the engineer, he writes the code and then we have a panel
Source: `L0139-L0143`
## Compliance measurement
> we're starting to see it now. So people that know how to orchestrate agents, people that know how to work with AI, they achieve significantly better outcomes.
Source: `L0243-L0247`
## Coverage metrics
> duplication goes down, code uh cognitive complexity goes down. So all metrics that we analyzed are actually now improving today with applying AI and and that wasn't always so like in the
Source: `L0332-L0336`
## Output evals
> On the instruction following, this is the rubrics, the metrics that are grounded specifically in what the skill is telling the agent how to do. So if you have your own internal design
Source: `L0422-L0426`
## Trajectory evals
> that unique flavor of how you as a company want your agents to be operating? So this is kind of a concrete example that we found that I found super interesting. So I use a lot of hugging
Source: `L0495-L0499`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |