Skip to content

aipoch/open-science / resources/skills/skill-creator

skill-creator

Verified

Create, revise, evaluate, publish, and improve Open-Science Skills through the native JavaScript host.skills composer. Use when the user wants a reusable workflow, an existing Skill changed, test cases or benchmarks for a Skill, or better Skill triggering.

scripts/aggregate-benchmark.js

View on GitHub
'use strict'
/* eslint-disable @typescript-eslint/no-require-imports, @typescript-eslint/explicit-function-return-type */

const { summarize } = require('./utils')

const METRICS = ['pass_rate', 'time_seconds', 'tokens', 'tool_calls', 'errors']

const summarizeConfiguration = (runs) =>
  Object.fromEntries(
    METRICS.map((metric) => [
      metric,
      summarize(runs.map((run) => Number(run.result?.[metric] ?? 0)).filter(Number.isFinite))
    ])
  )

const aggregateBenchmark = ({ metadata = {}, runs = [], notes = [] }) => {
  const configurations = [...new Set(runs.map((run) => run.configuration))]
  const runSummary = Object.fromEntries(
    configurations.map((configuration) => [
      configuration,
      summarizeConfiguration(runs.filter((run) => run.configuration === configuration))
    ])
  )

  const baseline = runSummary.without_skill ?? runSummary.old_skill
  if (runSummary.with_skill && baseline) {
    runSummary.delta = Object.fromEntries(
      METRICS.map((metric) => [metric, runSummary.with_skill[metric].mean - baseline[metric].mean])
    )
  } else {
    runSummary.delta = {}
  }

  return { schema_version: 1, metadata, runs, run_summary: runSummary, notes }
}

module.exports = { aggregateBenchmark, summarizeConfiguration }
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall