reply-team/jason-ai / skills/business/performance-analysis
performance-analysis
VerifiedPerformance analysis
Knowledge, not the enforced contract. This skill names operations from the vendor-neutral catalogue in
sdr-operations, which says what an SDR operation is. What this runtime validates and executes is the smaller set of JSON documents underdocs/contracts/operations/; where the two disagree, the contract is what decides.
Partly an expert skeleton. The diagnostic method is complete. The benchmark numbers below are explicitly heuristics carried over from the prototype and are marked
TODO(expert)— they need validation against real outbound data, by the experts who own this skill, before it leavesdraft. Treat them as rules of thumb, never as law. They are also this skill's own bands and nothing more: the contract returns no benchmark, industry average or portable score at all (M7), so a band here can never be confused with a figure something handed back.
Purpose
Turn raw figures into a diagnosis and one or two concrete actions. Read-only by default: nothing changes until the user approves a change. The value is not the numbers — the user can see those — it is correctly attributing a problem to deliverability, copy or targeting, because each has a different fix.
A number is not evidence until it carries its measurement envelope — when it was computed, the
window and timezone, window-or-cohort basis, the denominator's name and its count, and how
the fact was detected (M1). Half of the work below is refusing to diagnose on a figure that
arrives without one.
When to use / when NOT to use
- Use for performance questions, optimisation, and pause-or-fix decisions.
- Do NOT use to launch or change campaigns — that is
campaign-launch. - Do NOT use to work replies — that is
inbox-triage.
Prerequisites
- Enough volume to say anything. Below roughly a hundred delivered messages, say the sample
is too small and stop there rather than dressing noise as insight. The read itself will say
so too: a rate carries its denominator count and a typed insufficient-volume flag with the
threshold used (
M4), and that flag is the answer, not a caveat to write around. - Operations used, all
read, allauto, and none of them writes anything (M8):engagement.summarize,funnel.summarize,metric.describe,metric.compare,response_time.summarize,opportunity.summarize,engagement_event.list,sender.health,sender.summarize. Measurement is commonly rate-limited harder than everything else — sequential, cached, never polled.
There is one measurement read, not one per question. The metrics.* family is retired.
There is no account-overview operation, no per-campaign operation and no channel-efficiency
operation to look for: it is engagement.summarize, and the grouping is an argument —
engagement.summarize(scope, window, group_by=…). Anyone hunting for a per-campaign metrics
name will not find one, and inventing one is how a name that exists in a single playbook gets
born. The four purpose-built reads beside it are genuinely different questions, not groupings:
funnel.summarize for the whole ladder stage by stage with each stage's declared denominator,
opportunity.summarize for what came out of the far end and under which attribution model,
response_time.summarize for how fast we actually pick things up, and metric.compare for two
arms side by side.
Which groupings a measure actually carries is a read, not an assumption. The contract does
not publish a fixed list of group-by keys; metric.describe returns them per measure, along
with what the measure was mapped from and how far it can be trusted. Ask it before planning an
analysis around a grouping — including channel.
Planning guidance
Anchor the window first. Every figure is meaningless without the period it covers; agree last week, last month or a specific range with the user before pulling anything.
Define the measure before reading it. metric.describe first, because "reply rate" has no
shared definition and the commonest way an investigation ends is discovering that the two
figures being compared are two different measures. Cohort versus window is arithmetic, not
preference (M3): a cohort three days into a fourteen-day cadence has had two of seven touches
and is not comparable to a mature one.
Compare campaigns to each other, not only to published benchmarks. The user's own
best-performing campaign is a far better yardstick than an industry average, because it shares
their audience, domain and voice — and because a benchmark is one of the things the contract
deliberately declines to hand back (M7), so the comparison has to be built from the user's own
arms whether or not anyone wanted it that way.
The three causes are three groupings of one read. Deliverability is the sender axis
(sender.health, sender.summarize). Targeting is engagement.summarize grouped by
provenance, verification verdict, enrichment source or import batch — those are group-by keys
precisely so this axis is diagnosable rather than blamed on the copy (M6). Copy is the same
read grouped by step. Reaching for a different operation per cause is the wrong instinct.
Order is not preference. Each check can invalidate the question the next one would answer,
so running them out of order produces a confident wrong answer. sdr-operations carries the
full worked ordering as the chain Diagnosing a drop in reply rate; read it when the question
is a drop rather than a routine read.
Execution guidance
-
Define the measure (
metric.describe) — what it means here, what it was mapped from, what it can be grouped by, how far to trust it. -
Account overview (
engagement.summarize(scope, window)) for the agreed window. Read the did-not-happen counts first — deferred, cap-skipped, blackout-skipped, refused, throttled are first-class facts by reason (M5). Fewer replies is often fewer sends, and the rate may not have moved at all. -
Per campaign — the same operation with a grouping (
engagement.summarize(group_by=campaign)), then look closer at the outliers, best and worst. Per-channel and per-step are the same call again with a different key. -
Check the denominator and the volume flag before diagnosing anything (
M4). This step ends more investigations than any other, and ending one honestly is a result. -
Read what is current before you name anything that failed. A failure is a fact about a moment, and "what is wrong now" is a claim about today. Read where the campaign's sends are routed now and the latest outcomes of its work, newest first, and date every failure you mean to report against them. A failure from before the sends had a route, one a later success has superseded, and a role whose agent program was not installed yet (
host_not_available) are history: say so as history, with its date, and leave it out of what is wrong. Only a failure that is still the latest outcome, where the sends still have no route, is current. A figure the reads give you stands as given. Ifcampaign activitysays one person was contacted, the summary does not open with "nothing has gone out yet"; if its sends succeeded after the failure, a line that says nothing can be sent is wrong whatever the older item says.With Jason, the same reads are:
jason route resolve --campaign cmp_01JB6K8TQ2W9V4MZ0C3Y7H5NRD --operation message.send --channel email --human jason workitem list --campaign cmp_01JB6K8TQ2W9V4MZ0C3Y7H5NRD --status failed --status expired --order finished --limit 20 jason workitem list --campaign cmp_01JB6K8TQ2W9V4MZ0C3Y7H5NRD --status succeeded --kind provider_op --order finished --limit 20 jason campaign activity cmp_01JB6K8TQ2W9V4MZ0C3Y7H5NRD --since 2026-10-01T00:00:00Z --until nowA campaign can send email through one plugin and LinkedIn messages through another, so the route is asked per channel: name the channel the figures are about,
emailhere, and ask again with--channel linkedinfor the other.A plain
workitem listis oldest first and stops at a hundred, so a long campaign would show only its oldest items;--order finishedputs the newest first, twenty from each list. The succeeded items are provider operations of every kind, so the latest finished send is the newestmessage.sendamong them. Compare each failed item'sfinished_atwith it: a failed item older than a later success on the same route is not what is wrong now. -
Diagnose. Heuristic bands for cold outbound — TODO(expert): validate every number here: delivered ≥ ~95%, opened ~40–60%, replied ~2–8%, interested ~10–30% of replies.
- Deliverability problem — bounces above ~3%, or delivered below ~95%. The cause is
list quality or sender health, not the message. Check
sender.healthandsender.summarize— each failure class, complaint rate and deferral carries its own publisher, denominator and window, and no composite "health score" is manufactured from them (F6) — then go tosending-guardrails. - Copy problem — delivery and opens healthy, replies below ~1%. The plumbing works and
the message is not earning a response. Confirm it with
engagement.summarize(group_by=step)— which step lost the replies — before believing it. - Targeting problem — replies exist but almost nobody is interested. The message is
reaching people it should not reach: an audience mismatch, so go back to
audience-building. Prove it rather than asserting it, withengagement.summarize(group_by=provenance), then by verification verdict, enrichment source or import batch (M6). One call usually settles it: replies from the June import at 3.4% against a purchased file at 0.9% is the whole answer. - Caveat on opens. Open tracking is distorted by privacy proxies that fetch images
automatically or block them entirely, so open rate is the weakest signal in the set.
Never diagnose on opens alone; prefer delivered and replied.
metric.describeis the operation that carries this — what the open measure was mapped from, its detection method and its trust — and it exists so the discount is read rather than remembered. Note what is not on offer at all: there is no inbox-placement rate and no "did they read it" measure, because neither is measurable (M2), and a placement test is typed as simulated placement, never as an inbox rate.- TODO(expert): how much to discount open rate, and whether to report it at all.
- Correlation is not causation. When the data is thin, say so — an honest "we cannot tell yet" is worth more than a confident wrong answer.
- Deliverability problem — bounces above ~3%, or delivered below ~95%. The cause is
list quality or sender health, not the message. Check
-
Widen only if the question asks for it.
funnel.summarizewhen the question is where the ladder leaks rather than how one campaign is doing;opportunity.summarizewhen it is what the outreach was actually worth, withattribution.explainfor why an outcome was credited where it was;response_time.summarizewhen the loss is speed of follow-up, which no copy fix reaches. Where the user wants the rows behind a number, that isengagement_event.list. -
Comparing two arms is
metric.compare. It returns the counts, the interval around each rate, the sample size the comparison needs, an underpowered flag and the number of reads taken — and it never names a winner (M7). Report it that way. Re-reading until a difference appears is exactly the failure an agent on an hourly schedule will produce. -
Recommend one or two actions, highest impact first: pause the worst performer (reversible, cheap), rewrite an opener modelled on a campaign that is working, clean the bad addresses out of a list, spread volume across another sender.
-
Only on explicit approval, act — with one deliberate exception. Stopping is protective:
campaign.pauseandsender.pausederiveautoand never wait for permission, because waiting is the destructive choice when something is actively going wrong. Restarting is not symmetric:campaign.resumeandsender.resumeareconfirm_once, and that asymmetry is deliberate rather than an oversight. Everything else — starting, removing, changing content — inherits the gates incampaign-launch.
What no operation covers. Nothing in the contract records a diagnosis or a recommendation;
those are this skill's output and they live in the report, kept by whichever runtime the user
has. And no operation returns a benchmark, an industry average, a portable score or a named
winning variant — that is a refusal on the record (M7), not an absence to be worked around by
quoting one from elsewhere as though it came from the data.
Validation
Every recommendation quotes the actual figures with their envelope — window, basis,
denominator count, as-of (M1) — and a rate offered without its denominator count is not a
finding. Where a comparison spans two readings, confirm the detection method did not change
inside the window; a change of method is a mandatory caveat, not a footnote. After any approved
change, re-read the affected campaign's state (campaign.get) to confirm the change took
effect. If findings confirm or contradict guidance in another skill, that is evidence worth
recording — see the reporting conventions of whichever runtime the user has.
Reporting
A compact table: campaign / delivered / opened % / replied % / interested, each rate with its denominator count. One or two sentences of diagnosis per problem campaign, naming which of the three causes it is and why. Actions taken with identifiers. A single highest-impact next step. Always the window and the basis.
Say not measurable where that is the honest answer (M2) rather than substituting the
nearest available proxy. Where the user wants the underlying rows, engagement_event.list;
where they want a file, export.request is confirm_once and raised on purpose, because an
extract moves personal data out of the system that enforces its handling rules.
Failure modes
- Over-reading a small sample. The commonest way analysis does harm: a recommendation built on twelve sends sends the user off in the wrong direction.
- Diagnosing on open rate, which is the least trustworthy figure available.
- Reporting numbers without a window, which makes them unusable and uncomparable.
- Quoting a rate without its denominator count (
M4), which turns a direction into a claim. - Hunting for an operation per question. The retired
metrics.*family had one name per grouping; this contract has one read and a grouping argument. Looking for the missing name and inventing it is worse than askingmetric.describewhat the grouping keys are. - Reporting an old failure as current. A send that failed before its route was set, or before a later one succeeded, is history; naming it as the problem, or letting it outweigh a figure the reads gave, sends the user to repair something that already works.
- Reading a drop in reply count as a drop in reply rate without checking the
did-not-happen counts first (
M5). - Naming a winning variant because one arm is ahead on the screen (
M7). - Parallelising measurement reads and hitting rate limits, then retrying harder.
- Recommending five things. Two get done; five get ignored.
- Treating a benchmark as a target. These bands are rules of thumb pending validation, and a campaign below one is not automatically broken.
Safety
Read-only until the user approves a change. Pausing is the protective direction and the
recommended first move when something is actively going wrong: it derives auto and does not
wait, while resuming is confirm_once. Never bulk-delete contacts as an "optimisation" — that
destroys data to improve a percentage. An export moves personal data out of the system and is
gated for that reason, not for its cost. Full rules: approval-boundaries.
Related skills
sdr-operations— the measurement family, the group-by discipline, and the worked chain for diagnosing a drop in reply rate.approval-boundaries— why stopping isautoand restarting is not.sending-guardrails— where a deliverability diagnosis goes next.audience-building— where a targeting diagnosis goes next.campaign-launch— applying fixes that involve new or changed campaigns.
Changelog
- 3.0.0 (2026-08-10): migrated to contract 2.0.0. The retired
metrics.*family is replaced byengagement.summarizewith the grouping as an argument — the account-overview, per-campaign and channel-efficiency reads were three names and are now one call with three groupings, said explicitly so the missing names are not reinvented. Namesfunnel.summarize,metric.compare,metric.describe,response_time.summarize,opportunity.summarize,engagement_event.listandsender.summarize;metric.describenow carries the open-rate discount, which was previously prose with nowhere to live. Corrects the approval model: pausing derivesautounder the protective floor and is no longer described as confirmed, while resuming isconfirm_once. Adds the measurement envelope, the denominator-count rule, the did-not-happen counts and the refusal to name a winner as cited invariants rather than restated advice, and states what the contract does not cover. Every benchmark band and all three expert markers are unchanged. - 2.0.0 (2026-07-30): renamed from
analyze-performanceand rewritten vendor-neutral for the business pack — provider reporting endpoints and scopes moved out to the provider's own skills. Diagnostic method and heuristic bands preserved; open-rate caveat added; every benchmark explicitly marked for expert validation. - 1.0.0 (2026-07-27): ported from the prototype workflow as
analyze-performance.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |