Diogenesoftoronto/keating / .agents/skills/ax-learn
ax-learn
VerifiedThis skill helps an LLM generate correct AxLearn code using @ax-llm/ax. Use when the user asks about self-improving agents, trace-backed learning, feedback-aware updates, or AxLearn modes.
AxLearn Codegen Rules (@ax-llm/ax)
Use this skill to generate AxLearn code that matches the current API.
Core Model
AxLearnwraps anAxGen.teacheris for judging, synthesis, and reflection.runtimeAIis the model being improved.forward()andstreamingForward()are inference-time APIs and auto-log traces when tracing is enabled.optimize()is offline learning.applyUpdate()is a bounded update API forcontinuousandplaybookmodes.ready()should be awaited before assuming checkpoints have been restored.improvementis the score delta from the previous/restored state.
Required Inputs
- Always provide
name. - Always provide
storage. - Always provide
teacher. - Always provide
runtimeAIif you calloptimize()orapplyUpdate().
Modes
batch: offline prompt learning only.continuous: offline optimization plus bounded feedback-awareapplyUpdate(...).playbook: structured context/playbook learning plusapplyUpdate(...).
Preferred Construction
import {
AxLearn,
ax,
ai,
type AxCheckpoint,
type AxStorage,
type AxTrace,
} from '@ax-llm/ax';
const storage: AxStorage = {
save: async (_name, _item) => {
// persist trace/checkpoint
},
load: async (_name, _query) => {
// return traces/checkpoints
return [];
},
};
const teacher = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
});
const runtimeAI = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
});
const gen = ax(`
customerQuery:string "User message" ->
supportReply:string "Agent reply"
`);
const agent = new AxLearn(gen, {
name: 'support-bot-v1',
storage,
teacher,
runtimeAI,
mode: 'continuous',
budget: 12,
examples: [
{
customerQuery: 'Where is my order?',
supportReply: 'Your order is in transit and should arrive in 2 days.',
},
{
customerQuery: 'I need a refund.',
supportReply: 'I can help with that. Please share your order number.',
},
],
generateExamples: false,
});
await agent.ready();
Runtime Pattern
const prediction = await agent.forward(runtimeAI, {
customerQuery: 'My package is late.',
});
const traces = await agent.getTraces({ limit: 1 });
if (traces[0]) {
await agent.addFeedback(traces[0].id, {
score: 0,
label: 'needs-empathy',
comment: 'Acknowledge the frustration more directly.',
});
}
Offline Optimization
const result = await agent.optimize({
// Optional overrides
budget: 20,
});
console.log(result.mode);
console.log(result.score);
console.log(result.improvement);
console.log(result.checkpointVersion);
result.improvement is the gain relative to the prior/restored score.
Continuous Update
Use applyUpdate(...) only in continuous or playbook mode.
- In
continuousmode,examplemay be input-only. predictionis the observed runtime output being critiqued.- If
exampleincludes expected output fields, that expected-output row stays eligible for scored optimization. - The observed
predictionrow is feedback/reflection context, not a scored train/validation row by itself. - Feedback-bearing scored examples should stay in the training pool when non-feedback rows can fill validation.
- In
playbookmode,getInstruction()returns the active composed prompt.
const update = await agent.applyUpdate({
example: {
customerQuery: 'My package is late.',
},
prediction,
feedback: {
score: 0,
label: 'needs-empathy',
comment: 'Acknowledge the frustration more directly.',
},
});
Playbook Mode
- Use
mode: 'playbook'when the learned artifact should be structured guidance, not just an instruction tweak. - Playbook checkpoints restore through
ready(). applyUpdate(...)in playbook mode performs an online structured update.getInstruction()should be treated as the active composed runtime prompt, even before optimization if the base prompt lives in the signature description.artifact.playbookSummaryshould match the persisted checkpointstate.artifactSummary.
How Learning Data Is Used
examplesand usable traces become scored optimization rows.- Feedback stored with
addFeedback(...)becomes reflection feedback for later optimization. - In continuous updates,
example + prediction + feedbackis used as an observed feedback event. - Input-only update examples are useful for reflection, but they are not promoted into scored examples unless expected outputs are present.
Important Options
const agent = new AxLearn(gen, {
name: 'agent-id',
storage,
teacher,
runtimeAI,
mode: 'batch', // 'batch' | 'continuous' | 'playbook'
budget: 20,
metric: async ({ prediction, example }) => {
return prediction.supportReply === example.supportReply ? 1 : 0;
},
criteria: 'accuracy and tone',
judgeOptions: {},
examples: [],
useTraces: true,
generateExamples: false,
synthCount: 20,
validationSplit: 0.2,
continuousOptions: {
feedbackWindowSize: 25,
maxRecentTraces: 100,
updateBudget: 4,
},
playbookOptions: {
maxEpochs: 2,
},
onTrace: (trace) => {
console.log(trace.id);
},
onProgress: (progress) => {
console.log(progress.round, progress.score);
},
});
Result Shape
type AxLearnResult = {
mode: 'batch' | 'continuous' | 'playbook';
score: number;
improvement: number;
checkpointVersion: number;
stats: {
trainingExamples: number;
validationExamples: number;
feedbackExamples: number;
durationMs: number;
mode: 'batch' | 'continuous' | 'playbook';
};
state?: {
mode: 'batch' | 'continuous' | 'playbook';
instruction?: string;
baseInstruction?: string;
score?: number;
continuous?: {
feedbackTraceCount?: number;
lastUpdateAt?: string;
};
playbook?: Record<string, unknown>;
artifactSummary?: Record<string, unknown>;
};
artifact?: {
playbook?: Record<string, unknown>;
playbookSummary?: {
feedbackEvents: number;
historyBatches: number;
bulletCount: number;
updatedAt?: string;
};
lastUpdateAt?: string;
feedbackExamples?: number;
};
};
Storage Notes
AxStorage.save(name, item)receives either a trace or checkpoint.AxStorage.load(name, query)should return arrays of traces or checkpoints.- Checkpoints may be returned unsorted.
AxLearnrestores the newest one client-side.
Do This
- Use
runtimeAIexplicitly. - Await
ready()before relying on restored state. - Run
optimize()off the hot path. - Use
continuousmode when you want bounded feedback-aware updates. - Use
playbookmode when you want persistent structured guidance. - Pass the real observed model output as
predictioninapplyUpdate(...). - Treat
getInstruction()in playbook mode as the live composed prompt, not just the raw base instruction.
Avoid This
- Do not assume
teacheris the optimized runtime model. - Do not call
applyUpdate()inbatchmode. - Do not claim feedback affects learning unless you are storing it with
addFeedback(...)or passing it toapplyUpdate(...). - Do not assume checkpoints load synchronously in the constructor.
- Do not treat
predictionas the gold answer in continuous updates.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 468k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |