Skip to content

northseadl/norix-skills / llm-agent-dev

llm-agent-dev

Unverified

LLM Agent engineering: pattern selection (12-mode matrix), data simulation, convergence iteration. Covers intent routing, function calling, ReAct, MCP, prompt chaining, guardrails, evaluation.

FileKindSize
SKILL.mdskill md17.6 KB
agents/openai.yamlconfig198 B
evals/evals.jsonconfig4.7 KB
references/convergence/iteration-protocol.mdmarkdown8.0 KB
references/eval-loop.mdmarkdown9.4 KB
references/feedback-protocol.mdmarkdown6.0 KB
references/intent-patterns.mdmarkdown5.8 KB
references/paradigms/context-engineering.mdmarkdown6.0 KB
references/paradigms/decision-tree.mdmarkdown2.7 KB
references/paradigms/memory-architecture.mdmarkdown4.0 KB
references/paradigms/observability.mdmarkdown4.5 KB
references/paradigms/pattern-matrix.mdmarkdown3.3 KB
references/paradigms/performance-benchmarking.mdmarkdown15.2 KB
references/paradigms/prompt-patterns.mdmarkdown5.0 KB
references/paradigms/safety-guardrails.mdmarkdown5.5 KB
references/paradigms/tool-design.mdmarkdown2.5 KB
references/patterns/01-crex-rule-router.mdmarkdown6.8 KB
references/patterns/02-slot-filling.mdmarkdown7.1 KB
references/patterns/03-function-calling.mdmarkdown3.9 KB
references/patterns/04-workflow-agent.mdmarkdown5.0 KB
references/patterns/05-react-agent.mdmarkdown3.4 KB
references/patterns/06-router-expert.mdmarkdown3.2 KB
references/patterns/07-mcp-tool-protocol.mdmarkdown5.0 KB
references/patterns/08-handoff-delegation.mdmarkdown4.7 KB
references/patterns/09-prompt-chaining.mdmarkdown3.0 KB
references/patterns/10-parallelizing.mdmarkdown3.2 KB
references/patterns/11-orchestrator-worker.mdmarkdown4.8 KB
references/patterns/12-evaluator-optimizer.mdmarkdown5.5 KB
references/scenarios/coding-assistant.mdmarkdown2.0 KB
references/scenarios/content-generation.mdmarkdown2.7 KB
references/scenarios/customer-service.mdmarkdown2.6 KB
references/scenarios/ecommerce-shopping.mdmarkdown4.2 KB
references/scenarios/food-ordering.mdmarkdown3.1 KB
references/scenarios/knowledge-qa.mdmarkdown2.6 KB
references/simulation/data-generation.mdmarkdown8.1 KB
references/skill-templates.mdmarkdown8.0 KB

references/patterns/12-evaluator-optimizer.md

View on GitHub
# 模式 ⑫ Evaluator-Optimizer (评估-优化循环)

> 一个 LLM 生成输出,另一个评估并反馈,循环迭代直到达标
> 延迟 5-30s · 可预测性 ★★★★☆ · 适合需要高质量输出的生成任务

## 核心理念

双 LLM 协作: Generator 负责产出,Evaluator 负责打分和反馈。
循环直到 Evaluator 满意或达到最大迭代次数。

```
┌─────────────────────────────────────────────┐
│                                              │
│  Input → [Generator] → Output V1             │
│               ↑            │                  │
│               │            ▼                  │
│          ┌────┴────┐  [Evaluator]             │
│          │Feedback │       │                  │
│          │+ Score  │  Pass? ─── Yes → Final   │
│          └─────────┘       │                  │
│                       No ──┘                  │
│                   (max N iterations)          │
│                                              │
└─────────────────────────────────────────────┘
```

## 与 Pillar III 收敛迭代的关系

**本模式是单次请求内的微循环**:
- Evaluator-Optimizer 是在一次用户请求中,Generator 和 Evaluator 快速迭代
- Pillar III 收敛迭代是跨多个版本/配置的宏观优化

两者可叠加: Pillar III 可以用多个 Evaluator-Optimizer 配置做 A/B 测试。

## 适用场景

- 文案 / 营销内容生成(质量要求高)
- 代码生成 + 自动审查
- 摘要/翻译质量迭代
- 结构化数据提取(Schema 合规性检查)
- SQL 生成 + 语法/逻辑验证

## 实现

```typescript
interface EvalResult {
  passed: boolean;
  score: number;         // 0-5
  feedback: string;      // 具体改进建议
  issues: string[];      // 问题列表
}

interface EvalOptConfig {
  generatorModel: string;
  evaluatorModel: string;    // 推荐用更强的模型评估
  maxIterations: number;     // 建议 2-4
  passThreshold: number;     // 分数阈值 (如 4.0)
  evaluationCriteria: string; // 评估标准
}

class EvaluatorOptimizer {
  constructor(private config: EvalOptConfig) {}

  async run(input: string): Promise<{ output: string; iterations: number; finalScore: number }> {
    let currentOutput = '';
    let feedback = '';
    let iterations = 0;

    for (let i = 0; i < this.config.maxIterations; i++) {
      iterations++;

      // Generate (含反馈上下文)
      currentOutput = await this.generate(input, currentOutput, feedback);

      // Evaluate
      const evalResult = await this.evaluate(input, currentOutput);

      if (evalResult.passed) {
        return { output: currentOutput, iterations, finalScore: evalResult.score };
      }

      feedback = evalResult.feedback;
    }

    // 达到最大迭代 — 返回最后版本
    return { output: currentOutput, iterations, finalScore: -1 };
  }

  private async generate(
    originalInput: string,
    previousOutput: string,
    feedback: string,
  ): Promise<string> {
    const messages: Message[] = [
      { role: 'user', content: originalInput },
    ];

    if (previousOutput && feedback) {
      messages.push({
        role: 'assistant',
        content: previousOutput,
      });
      messages.push({
        role: 'user',
        content: `评审反馈:\n${feedback}\n\n请根据反馈改进输出。`,
      });
    }

    const response = await llm.chat({
      model: this.config.generatorModel,
      messages,
      temperature: 0.3,  // 稍有创造性
    });

    return response.content;
  }

  private async evaluate(input: string, output: string): Promise<EvalResult> {
    const response = await llm.chat({
      model: this.config.evaluatorModel,
      messages: [{
        role: 'user',
        content: `你是质量评审专家。评估以下输出是否满足标准。

## 原始需求
${input}

## 当前输出
${output}

## 评估标准
${this.config.evaluationCriteria}

## 输出格式 (仅 JSON)
{
  "score": 4.5,
  "passed": true,
  "feedback": "整体质量好,但可以改进...",
  "issues": ["问题1", "问题2"]
}`,
      }],
      response_format: { type: 'json_object' },
      temperature: 0,
    });

    const result = JSON.parse(response.content);
    result.passed = result.score >= this.config.passThreshold;
    return result;
  }
}
```

## 示例: 营销文案迭代

```typescript
const copywriter = new EvaluatorOptimizer({
  generatorModel: 'gpt-4o',
  evaluatorModel: 'claude-sonnet-4-20250514',
  maxIterations: 3,
  passThreshold: 4.0,
  evaluationCriteria: `
    1. 情感吸引力 (1-5): 是否让人产生购买欲?
    2. 信息完整度 (1-5): 是否包含关键卖点?
    3. 品牌一致性 (1-5): 是否符合品牌调性?
    4. 行动号召 (1-5): 是否有清晰的 CTA?
    5. 合规性 (1-5): 是否避免虚假宣传?
    总分 = 平均分。通过阈值 = 4.0。
  `,
});

const result = await copywriter.run('为新款运动鞋写一段小红书种草文案');
// → 经过 2-3 轮迭代,输出高质量文案
```

## 成本控制

- 第一轮用低成本模型快速产出草稿
- Evaluator 始终用强模型(质量审查不能省)
- 设置严格的 maxIterations 上限(建议 ≤ 4)
- 大多数请求应在 2 轮内收敛(如果不能,说明 prompt 设计有问题)
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall