Skip to content

jscraik/Agent-Skills / Plugins/aidevcon/skills/talk-kushwaha-benchmarking-agent-era

talk-kushwaha-benchmarking-agent-era

Verified

Use when the user asks about Amit Kushwaha's AI Native DevCon talk on benchmarking agent-era systems, measuring performance beyond single LLM calls, inference, workflow complexity, tool use, and real-world workloads.

FileKindSize
SKILL.mdskill md2.4 KB
agents/openai.yamlconfig142 B
outline.mdmarkdown1.2 KB
quote.mdmarkdown2.0 KB
transcript.mdmarkdown38.3 KB
# Quotes -- Benchmarking the Agent Era: Measuring Performance Beyond the LLM

Short excerpts selected from the transcript for grounding answers. Preserve transcript artifacts when quoting.

## Agent-era benchmarking

> So you're going to be talking about benchmarking the benchmarking the agent era and you're ready to go. So I will leave you to it.

Source: `L0007-L0011`

## Inference performance

> three things. One, what does the agentic benchmark look agentic workload looks like? Second, what are the optimizations that already exist to run these agentic workloads efficiently? And third, what

Source: `L0032-L0036`

## Workflow complexity

> these applications were not using tools. Now in the agentic workloads what we are seeing is dozens of turns and I'll give you concrete examples of what what I mean. U the LLMs are getting called

Source: `L0060-L0064`

## Tool calls

> talk will mostly focus on the coding agent side of things. This workload might diff look different if you're talking about different kinds of agent but here the focus is primarily on the

Source: `L0135-L0139`

## Context length

> replicas of the same model that you're using in your agentic workloads. So turn one end up on replica A. If you do a roundroin kind of a setup, the turn two might end up on some other replica, but

Source: `L0189-L0193`

## Real-world workload measurement

> trajectory which is completely different from what we see in agentic workloads on the agentic workloads. Now what we miss there's no fixed shape as I saw the trajectory keeps growing as you are

Source: `L0318-L0322`

## Agent-era benchmarking

> So all this slide is trying to say in the agentic workload you can't just talk about mean median you have to look at distributions you have to talk about metrics in terms of distributions in

Source: `L0379-L0383`

## Inference performance

> representing your workload correctly especially in agentic workloads those gray regions where this stuff is being run on CPU is very important and that kind of helps you getting the right

Source: `L0500-L0504`

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall