Skip to content

getaero-io/gtm-eng-skills / engineering/warm-intro-scoring

warm-intro-scoring

Verified

FileKindSize
.gitignoreother121 B
GUIDE.mdmarkdown4.7 KB
LICENSEother1.1 KB
README.mdmarkdown8.0 KB
SKILL.mdskill md327 B
TESTING.mdmarkdown8.9 KB
UPSTREAM.mdmarkdown1.3 KB
assets/company-research.play.tsscript1021 B
assets/customer-db.sqlother2.7 KB
assets/evaluation-example.jsonconfig4.6 KB
assets/example.jsonconfig6.9 KB
assets/portfolio-focus.weights.jsonconfig32 B
assets/review.htmlother7.4 KB
assets/tuning-example.jsonconfig1.2 KB
assets/tuning.htmlother30.1 KB
check_all.pyscript1.4 KB
package-manifest.jsonconfig9.8 KB
plays/README.mdmarkdown18.2 KB
plays/check-report.pyscript1.8 KB
plays/check_all.pyscript1.6 KB
plays/check_parity.pyscript6.0 KB
plays/collect.play.tsscript4.0 KB
plays/collection.tsscript1.9 KB
plays/core.tsscript44.2 KB
plays/evaluation-contract.mdmarkdown3.4 KB
plays/evaluation.tsscript13.8 KB
plays/features.play.tsscript4.7 KB
plays/features.tsscript22.6 KB
plays/generatedtemplate.tsscript38.1 KB
plays/guardrails.tsscript948 B
plays/legacy.tsscript9.7 KB
plays/merge-payloads.pyscript8.6 KB
plays/parity-driver.tsscript995 B
plays/parity-evaluation.pyscript8.5 KB
plays/parity-reports.pyscript2.3 KB
plays/prepare-inputs.pyscript6.5 KB
plays/quality.tsscript3.8 KB
plays/read-sources.play.tsscript3.4 KB
plays/report-cli.tsscript590 B
plays/report.play.tsscript2.9 KB
plays/report.tsscript3.1 KB
plays/reports.tsscript5.7 KB
plays/research.play.tsscript1.6 KB
plays/review-uncertainty.play.tsscript5.5 KB
plays/score.play.tsscript2.3 KB
plays/snapshots.play.tsscript4.2 KB
plays/strict-json.tsscript1.8 KB
plays/sync-template.pyscript849 B
plays/uncertainty.tsscript4.9 KB
plays/validation.play.tsscript3.6 KB
references/adversarial-review.mdmarkdown7.0 KB
references/batch-collection.mdmarkdown5.1 KB
references/company-network.mdmarkdown5.8 KB
references/competitive-review.mdmarkdown8.8 KB
references/evaluation.mdmarkdown8.5 KB
references/evaluator-validation.mdmarkdown3.4 KB
references/evaluator.mdmarkdown7.9 KB
references/external-data.mdmarkdown4.0 KB
references/factor-test-matrix.mdmarkdown4.6 KB
references/factors.mdmarkdown6.1 KB
references/intro-drafts.mdmarkdown1.6 KB
references/pipeline.mdmarkdown7.6 KB
references/readable-output.mdmarkdown5.4 KB
references/research-brief.mdmarkdown2.4 KB
references/sources.mdmarkdown7.5 KB
references/tuning.mdmarkdown14.6 KB
scripts/audit_history.pyscript5.1 KB
scripts/check.pyscript5.1 KB
scripts/compare_versions.pyscript1.9 KB
scripts/evaluate.pyscript22.5 KB
scripts/package-skill.pyscript1.9 KB
scripts/portfolio.pyscript3.7 KB
scripts/quality.pyscript5.3 KB
scripts/recipe-demo.tsscript2.6 KB
scripts/render.pyscript1.6 KB
scripts/score.pyscript9.6 KB
scripts/temporal.pyscript2.2 KB
scripts/test_ask_handoff.pyscript2.1 KB
scripts/test_audit_history.pyscript2.7 KB
scripts/test_handoff_evaluate.pyscript9.1 KB
scripts/test_portfolio.pyscript3.6 KB
scripts/test_quality.pyscript3.1 KB
scripts/test_temporal.pyscript1.3 KB
scripts/test_tuning.pyscript14.1 KB
scripts/test_tuning_browser.cjsscript7.7 KB
scripts/test_version_parity.pyscript1.0 KB
scripts/tuning.pyscript10.9 KB
tests/adversarial.test.tsscript8.2 KB
tests/browser_e2e.cjsscript1.9 KB
tests/cloud_e2e.pyscript6.9 KB
tests/e2e-fixture.jsonconfig3.6 KB
tests/feature_parity.pyscript2.4 KB
tests/features.test.tsscript8.3 KB
tests/test_prepare_inputs.pyscript8.6 KB
tests/uncertainty.test.tsscript2.5 KB
vendor-manifest.jsonconfig748 B

references/evaluator-validation.md

View on GitHub
# Evaluator validation record

All examples in this record are fictional. These checks assess evaluator behavior, not intro success.

## Documentation RED → GREEN

Pressure scenario: an imminent demo, prior team investment and a product-owner demand for uplift. Two queries compare the same candidate sets; one has an unjudged route and post-cutoff evidence. Four asks were drafted, two sent, one replied, and one sent ask's observation window remains open.

The no-guidance control correctly held promotion, but reported:

> Reciprocal rank of first **known** relevant result, averaged over 2 queries | 1.00 | 0.75

> 1/2 sent asks received a reply (50%)

The first is a partial-label diagnostic rather than primary MRR. The second uses an immature denominator without knowing which ask replied. These failures motivated explicit eligibility rules in `evaluator.md`; the control did not demonstrate a tendency to approve deployment, so no such failure is claimed.

A fresh agent given the reference contract excluded the incomplete query, computed all three primary metrics as 1.0 on the single eligible query, and reported:

> Verified mature denominator: **unavailable** without dated sends, the fixed window, and observation dates.

> Fixed-window reply and meeting rates: **unavailable**.

It retained the temporal-leakage failure and declined an uplift claim despite deadline, sunk-cost and authority pressure. This is one control and one guided full scenario, not a five-repetition wording study or a statistically established agent-compliance rate. No behavioral-shaping wording comparison is claimed; the change is a metric eligibility reference contract.

## Reproduce the executable checks

Run the evaluator unit tests with the standard Python unittest runner from the repository:

```bash
python3 -m unittest discover -s skills/warm-intro-scoring/scripts -p 'test_evaluate.py'
python3 skills/warm-intro-scoring/scripts/check.py --repo .
```

The numerical fixture uses hand-derived ranking expectations. Tests exercise real parsing, metrics, gates and file output without provider mocks or paid calls. The demo dataset is synthetic and cannot establish predictive quality. See `evaluator.md` for human labeling, provenance and prospective trial requirements.

## Executed verification

The evaluator has 13 executable tests. Initial missing-module tests ran red before implementation; additional red tests preceded account clustering, separate scoring cutoffs, explicit-null split rejection and readable scorecard output. Fixed-window tests cover immature early replies, late replies, unknown outcomes and dated chronology. The complete repository run passed 129 tests and 78 subtests; the separate scorer suite passed 17 checks.

Independent adversarial review verified rejection of post-scoring features with timezone offsets, evaluated accounts outside the declared test split and declined ready paths. An outcome exactly at its horizon counted; one microsecond later did not. The reviewer identified the initially missing event-horizon rule; both contract and runner now enforce it.

Desktop/mobile browser checks passed for the two tables, expandable metadata, absence of page overflow and script errors. The fixture scorecard is explicitly insufficient evidence: one eligible query out of three and one complete account. Neither this fictional result nor the pressure-test pass is a real-world accuracy estimate.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall