Skip to content

northseadl/norix-skills / llm-agent-dev

llm-agent-dev

Unverified

LLM Agent engineering: pattern selection (12-mode matrix), data simulation, convergence iteration. Covers intent routing, function calling, ReAct, MCP, prompt chaining, guardrails, evaluation.

FileKindSize
SKILL.mdskill md17.6 KB
agents/openai.yamlconfig198 B
evals/evals.jsonconfig4.7 KB
references/convergence/iteration-protocol.mdmarkdown8.0 KB
references/eval-loop.mdmarkdown9.4 KB
references/feedback-protocol.mdmarkdown6.0 KB
references/intent-patterns.mdmarkdown5.8 KB
references/paradigms/context-engineering.mdmarkdown6.0 KB
references/paradigms/decision-tree.mdmarkdown2.7 KB
references/paradigms/memory-architecture.mdmarkdown4.0 KB
references/paradigms/observability.mdmarkdown4.5 KB
references/paradigms/pattern-matrix.mdmarkdown3.3 KB
references/paradigms/performance-benchmarking.mdmarkdown15.2 KB
references/paradigms/prompt-patterns.mdmarkdown5.0 KB
references/paradigms/safety-guardrails.mdmarkdown5.5 KB
references/paradigms/tool-design.mdmarkdown2.5 KB
references/patterns/01-crex-rule-router.mdmarkdown6.8 KB
references/patterns/02-slot-filling.mdmarkdown7.1 KB
references/patterns/03-function-calling.mdmarkdown3.9 KB
references/patterns/04-workflow-agent.mdmarkdown5.0 KB
references/patterns/05-react-agent.mdmarkdown3.4 KB
references/patterns/06-router-expert.mdmarkdown3.2 KB
references/patterns/07-mcp-tool-protocol.mdmarkdown5.0 KB
references/patterns/08-handoff-delegation.mdmarkdown4.7 KB
references/patterns/09-prompt-chaining.mdmarkdown3.0 KB
references/patterns/10-parallelizing.mdmarkdown3.2 KB
references/patterns/11-orchestrator-worker.mdmarkdown4.8 KB
references/patterns/12-evaluator-optimizer.mdmarkdown5.5 KB
references/scenarios/coding-assistant.mdmarkdown2.0 KB
references/scenarios/content-generation.mdmarkdown2.7 KB
references/scenarios/customer-service.mdmarkdown2.6 KB
references/scenarios/ecommerce-shopping.mdmarkdown4.2 KB
references/scenarios/food-ordering.mdmarkdown3.1 KB
references/scenarios/knowledge-qa.mdmarkdown2.6 KB
references/simulation/data-generation.mdmarkdown8.1 KB
references/skill-templates.mdmarkdown8.0 KB

references/paradigms/performance-benchmarking.md

View on GitHub
# 性能评估与优化 (Performance Benchmarking & Optimization)

> Agent 系统的延迟剖析、模型基线测试与极致速度优化策略

## 两层评估体系

```
┌──────────────────────────────────────────────────────────────┐
│              Agent 性能评估金字塔                               │
│                                                              │
│              ┌────────────┐                                  │
│              │  Layer 2   │  Agent 管道延迟剖析               │
│              │  Pipeline  │  每环节 P50/P95/P99              │
│             ┌┴────────────┴┐                                 │
│             │   Layer 1    │  模型性能基线                    │
│             │   Baseline   │  TTFT + TPS + 上下文/缓存影响    │
│             └──────────────┘                                 │
└──────────────────────────────────────────────────────────────┘
```

---

## Layer 1: 模型性能基线

> 在标准环境下测量模型裸性能,建立比较基准

### 1.1 核心指标

| 指标 | 全称 | 定义 | 测量方法 |
|------|------|------|---------|
| **TTFT** | Time to First Token | 从请求发出到收到第一个 token 的时间 | 流式响应首 chunk 时间戳 - 请求发出时间戳 |
| **TPS** | Tokens Per Second | 平均 token 生成速率 | (total_output_tokens - 1) / (last_token_time - first_token_time) |
| **E2E Latency** | End-to-End | 从请求到完整响应 | response_received - request_sent |
| **TPOT** | Time Per Output Token | 每 token 平均生成时间 | 1 / TPS |

### 1.2 基线测试矩阵

```typescript
interface BaselineBenchmark {
  // 测试维度
  dimensions: {
    model: string[];           // 候选模型列表
    contextLength: number[];   // [500, 2000, 8000, 32000, 128000]
    outputLength: number[];    // [50, 200, 500, 1000]
    cacheMode: ('cold' | 'warm' | 'prompt_cache')[];
    concurrency: number[];     // [1, 5, 10, 20]
  };

  // 每组测试重复次数(统计显著性)
  runsPerConfig: number;       // 建议 ≥ 5

  // 测试 prompt 标准化
  standardPrompts: {
    classification: string;    // 意图分类任务 (短输出)
    generation: string;        // 内容生成任务 (长输出)
    reasoning: string;         // 推理任务 (中等输出)
  };
}
```

### 1.3 缓存影响测试

```
冷启动 (Cold):
  首次请求,无任何缓存
  → 测量: TTFT_cold, TPS_cold

热启动 (Warm):
  连接已建立,TCP/TLS 复用
  → 测量: TTFT_warm, TPS_warm
  → 预期: TTFT 降低 30-50%

Prompt Cache (厂商级):
  System Prompt 被厂商缓存
  → 测量: TTFT_cached, TPS_cached, input_cost_cached
  → 预期: TTFT 降低 50-80%, 成本降低 50-90%
  → 适用: OpenAI/Anthropic/DeepSeek 的 prompt caching
```

### 1.4 上下文长度影响

```
测试: 在 context_length = [500, 2000, 8000, 32K, 128K] 下测量

预期趋势:
  TTFT:  随 context 增长线性/亚线性增长
  TPS:   通常不受 input length 影响(受 output length 影响)
  Cost:  随 context 线性增长

关键发现:
  • 找到 TTFT 陡增的拐点(通常在 32K-64K)
  • 评估 Prompt Cache 在大 context 下的收益
  • 为 Context Budget 策略提供数据依据
```

### 1.5 基线测试脚本

```typescript
// benchmark/baseline.ts

interface BenchmarkResult {
  model: string;
  contextLength: number;
  outputLength: number;
  cacheMode: string;
  concurrency: number;

  // 延迟指标 (ms)
  ttft: { p50: number; p95: number; p99: number; mean: number };
  tps: { p50: number; p95: number; p99: number; mean: number };
  e2eLatency: { p50: number; p95: number; p99: number; mean: number };

  // 成本指标
  inputTokens: number;
  outputTokens: number;
  cachedTokens: number;
  costUSD: number;

  // 元信息
  timestamp: string;
  region: string;
  runs: number;
}

async function runBaseline(config: BaselineBenchmark): Promise<BenchmarkResult[]> {
  const results: BenchmarkResult[] = [];

  for (const model of config.dimensions.model) {
    for (const ctxLen of config.dimensions.contextLength) {
      for (const outLen of config.dimensions.outputLength) {
        for (const cache of config.dimensions.cacheMode) {
          for (const conc of config.dimensions.concurrency) {
            const measurements: RawMeasurement[] = [];

            for (let i = 0; i < config.runsPerConfig; i++) {
              const m = await measureSingleCall({
                model,
                contextLength: ctxLen,
                maxOutputTokens: outLen,
                cacheMode: cache,
                streaming: true,
              });
              measurements.push(m);
            }

            results.push(aggregateToPercentiles(measurements, {
              model, contextLength: ctxLen, outputLength: outLen,
              cacheMode: cache, concurrency: conc,
            }));
          }
        }
      }
    }
  }

  return results;
}

async function measureSingleCall(params: CallParams): Promise<RawMeasurement> {
  const requestTime = performance.now();
  let firstTokenTime: number | null = null;
  let lastTokenTime: number | null = null;
  let totalTokens = 0;

  const stream = await llm.chatStream({
    model: params.model,
    messages: buildMessages(params.contextLength),
    maxTokens: params.maxOutputTokens,
    temperature: 0,
    stream: true,
  });

  for await (const chunk of stream) {
    if (!firstTokenTime) firstTokenTime = performance.now();
    lastTokenTime = performance.now();
    totalTokens += countTokens(chunk.content);
  }

  return {
    ttft: firstTokenTime! - requestTime,
    totalDuration: lastTokenTime! - requestTime,
    generationDuration: lastTokenTime! - firstTokenTime!,
    outputTokens: totalTokens,
    tps: totalTokens > 1
      ? (totalTokens - 1) / ((lastTokenTime! - firstTokenTime!) / 1000)
      : 0,
    inputTokens: params.contextLength,
  };
}
```

### 1.6 基线报告格式

```
## 模型基线报告: {date}

### TTFT (Time to First Token)

| Model | 500 ctx | 2K ctx | 8K ctx | 32K ctx | Cache ON |
|-------|---------|--------|--------|---------|----------|
| GPT-4o-mini | 180ms | 220ms | 350ms | 680ms | 95ms |
| Claude Haiku | 150ms | 200ms | 310ms | 590ms | 80ms |
| Qwen-Turbo | 120ms | 160ms | 280ms | 520ms | N/A |
| DeepSeek V3 | 200ms | 250ms | 400ms | 750ms | 110ms |

### TPS (Tokens Per Second)

| Model | Short Output | Medium Output | Long Output |
|-------|-------------|--------------|------------|
| GPT-4o-mini | 85 | 82 | 78 |
| Claude Haiku | 95 | 90 | 85 |
| Qwen-Turbo | 110 | 105 | 98 |

### 结论
- 最低 TTFT: {model} @ {value}ms (cached)
- 最高 TPS: {model} @ {value} tok/s
- 推荐生产模型: {model} (综合 TTFT + TPS + 成本)
```

---

## Layer 2: Agent 管道延迟剖析

> 将 Agent 请求拆解到每个环节,测量并优化端到端延迟

### 2.1 管道阶段模型

```
用户请求 ─────────────────────────────────────────── 响应返回
    │                                                    │
    ▼                                                    │
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐  │
│网络传输│→│输入安全│→│意图分类│→│技能路由│→│技能执行│→│输出安全│→ │
│      │ │检查   │ │      │ │      │ │      │ │检查   │  │
│ T_net│ │T_guard│ │T_cls │ │T_route│ │T_exec│ │T_out │  │
└──────┘ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘  │
                                                         │
E2E = T_net_in + T_guard + T_cls + T_route + T_exec + T_out + T_net_out
```

### 2.2 逐环节基准

| 环节 | 预算 | 优化杠杆 | 测量方法 |
|------|------|---------|---------|
| **T_net** (网络) | ≤ 100ms | CDN/边缘部署 | ping/traceroute |
| **T_guard** (输入安全) | ≤ 5ms | 正则预编译 | 函数计时 |
| **T_cls: 缓存命中** | ≤ 2ms | 缓存预热 | 函数计时 |
| **T_cls: 规则短路** | ≤ 1ms | 规则优化 | 函数计时 |
| **T_cls: LLM 调用** | ≤ 500ms | 模型选择/Prompt Cache | 流式 TTFT |
| **T_route** (路由) | ≤ 1ms | Map.get() | 函数计时 |
| **T_exec: 数据查询** | ≤ 200ms | 索引/连接池 | DB 耗时 |
| **T_exec: LLM 增强** | ≤ 1000ms | 模型/并行 | 流式 TTFT |
| **T_out** (输出安全) | ≤ 3ms | 正则预编译 | 函数计时 |

**总预算: ≤ 2000ms (P95)**

### 2.3 管道 Profiler 实现

```typescript
class PipelineProfiler {
  private spans: Map<string, { start: number; end?: number }> = new Map();

  startSpan(name: string) {
    this.spans.set(name, { start: performance.now() });
  }

  endSpan(name: string) {
    const span = this.spans.get(name);
    if (span) span.end = performance.now();
  }

  getReport(): PipelineReport {
    const stages: StageMetric[] = [];
    let total = 0;

    for (const [name, span] of this.spans) {
      const duration = (span.end ?? performance.now()) - span.start;
      stages.push({ name, durationMs: Math.round(duration) });
      total += duration;
    }

    return {
      totalMs: Math.round(total),
      stages,
      bottleneck: stages.reduce((a, b) => a.durationMs > b.durationMs ? a : b),
      budgetStatus: total <= 2000 ? 'PASS' : 'FAIL',
    };
  }
}

// 使用
async function handleMessageWithProfiling(message: string, ctx: AgentContext) {
  const profiler = new PipelineProfiler();

  profiler.startSpan('input_guard');
  const guardResult = inputGuard.check(message);
  profiler.endSpan('input_guard');

  profiler.startSpan('classify');
  const intent = await classifier.classify(message);
  profiler.endSpan('classify');

  profiler.startSpan('route');
  const handler = router.resolve(intent.intent);
  profiler.endSpan('route');

  profiler.startSpan('execute');
  const response = await handler.execute(intent.params, ctx);
  profiler.endSpan('execute');

  profiler.startSpan('output_guard');
  const sanitized = outputGuard.sanitize(response);
  profiler.endSpan('output_guard');

  // 记录 profiler 结果到日志
  const report = profiler.getReport();
  log.info('pipeline_profile', report);

  return sanitized;
}
```

### 2.4 管道 Profiling 报告

```
## Agent 管道延迟报告

### P95 延迟分布 (N=1000 requests)

| 环节 | P50 | P95 | P99 | 预算 | 状态 |
|------|-----|-----|-----|------|------|
| input_guard | 1ms | 2ms | 3ms | 5ms | ✅ |
| classify (cache hit 35%) | 2ms | 3ms | 5ms | 2ms | ✅ |
| classify (shortcut 15%) | 0ms | 1ms | 1ms | 1ms | ✅ |
| classify (LLM 50%) | 280ms | 450ms | 620ms | 500ms | ✅ |
| route | 0ms | 0ms | 1ms | 1ms | ✅ |
| execute (data_query) | 35ms | 120ms | 280ms | 200ms | ✅ |
| execute (llm_enhance) | 650ms | 1100ms | 1500ms | 1000ms | ⚠ |
| output_guard | 1ms | 2ms | 3ms | 3ms | ✅ |
| **E2E** | **520ms** | **1680ms** | **2410ms** | **2000ms** | ⚠ |

### 瓶颈分析
- 主要瓶颈: execute.llm_enhance (占 65% 延迟)
- 优化建议: 考虑流式返回 / 切换到更快模型 / 预生成缓存
```

---

## 极致速度优化策略

### 策略 1: 分层缓存

```
L0: 正则短路          → 0ms   (Top 10 高频意图)
L1: 精确缓存命中      → 1ms   (近 5 分钟相同输入)
L2: 模糊缓存命中      → 2ms   (编辑距离 < 3 的输入)
L3: Prompt Cache      → 减少 50-80% TTFT
L4: 结果缓存          → FAQ 等不变内容,缓存完整响应
```

### 策略 2: 流式响应

```typescript
// 不等待完整响应,流式逐步返回
async function* streamResponse(message: string): AsyncGenerator<ResponseChunk> {
  // 1. 分类完成后立即返回 skeleton
  yield { type: 'skeleton', intent: intent.intent };

  // 2. 数据查询完成后返回结构化数据
  yield { type: 'data', content: queryResult };

  // 3. LLM 增强部分流式返回
  for await (const chunk of llm.stream()) {
    yield { type: 'text_delta', content: chunk };
  }
}
```

### 策略 3: 预测性预取

```typescript
// 基于意图历史预测下一个意图,提前加载数据
class PredictivePreloader {
  private transitionMatrix: Map<string, Map<string, number>>;

  predictNext(currentIntent: string): string | null {
    const transitions = this.transitionMatrix.get(currentIntent);
    if (!transitions) return null;
    // 返回概率最高的下一个意图
    return [...transitions.entries()].sort((a, b) => b[1] - a[1])[0]?.[0] ?? null;
  }

  async preload(predictedIntent: string) {
    // 在后台预加载数据
    const handler = router.resolve(predictedIntent);
    if (handler?.preloadable) {
      await handler.preload();
    }
  }
}
```

### 策略 4: 模型选择金字塔

```
速度优先:
  意图分类: Qwen-Turbo (TTFT ~120ms, TPS ~110)
  参数提取: 同上 (单次调用同时完成)
  LLM增强: Claude Haiku (TPS ~95)
  评估: Claude Sonnet (离线,不影响延迟)

质量优先:
  意图分类: GPT-4o-mini (TTFT ~180ms, 准确率更高)
  LLM增强: GPT-4o (质量最高)
  评估: Claude Sonnet + GPT-4o 交叉

混合模式:
  简单意图: 规则短路 (0ms)
  中等意图: Qwen-Turbo (120ms)
  复杂意图: GPT-4o-mini (180ms)
  → 根据意图复杂度动态选模型
```

### 策略 5: 并行化非阻塞环节

```typescript
// 输入安全检查和缓存查询可以并行
const [guardResult, cacheResult] = await Promise.all([
  inputGuard.check(message),
  cache.lookup(message),
]);

// 如果缓存命中,直接返回(跳过 LLM 调用和安全检查并行完成了)
if (cacheResult) return cacheResult;
if (!guardResult.safe) return blockedResponse;

// LLM 分类和数据预取可以并行
const [intent, preloadedData] = await Promise.all([
  classifier.classify(message),
  predictivePreloader.preloadIfConfident(message),
]);
```

### 策略 6: 连接池与复用

```typescript
// HTTP 连接复用(避免 TCP/TLS 握手开销)
const httpAgent = new Agent({
  keepAlive: true,
  maxSockets: 20,
  keepAliveMsecs: 30000,
});

// 数据库连接池
const dbPool = {
  maxConnections: 25,
  idleTimeoutMs: 30000,
  connectionTimeoutMs: 5000,
};
```

---

## 性能回归测试

### CI 集成

```yaml
# 每次部署前运行性能回归
performance-test:
  steps:
    - name: Run Pipeline Benchmark
      run: npx ts-node benchmark/pipeline.ts --runs 100
      env:
        BENCHMARK_MODE: true

    - name: Compare with Baseline
      run: npx ts-node benchmark/compare.ts \
        --current results/current.json \
        --baseline results/baseline.json \
        --max-regression 10%
      # 任何环节 P95 回退 > 10% → 阻断部署
```

### 性能 SLO

```typescript
interface PerformanceSLO {
  e2e: {
    p50: 500;      // ms
    p95: 2000;     // ms
    p99: 3000;     // ms
  };
  ttft: {
    p50: 200;      // ms
    p95: 500;      // ms
  };
  cacheHitRate: 0.30;     // ≥ 30%
  shortcutHitRate: 0.15;  // ≥ 15%

  // 回归保护
  maxRegressionPercent: 10; // 任何指标回退不超过 10%
}
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall