Skip to content

getaero-io/gtm-eng-skills / skills/deepline-scoring

deepline-scoring

Verified

Use when discovering niche signals, auditing ICP or won/lost evidence, rescoring accounts, or building account and lead scoring Plays. Triggers on fit scoring, engagement scoring, backtesting, holdout evaluation, predictive lift, external proxies, and scoring leakage. Skip pure outreach copy or contributor skill installation tasks.

FileKindSize
.gitignoreother26 B
SKILL.mdskill md12.4 KB
package.jsonmanifest114 B
references/account-website-features.mdmarkdown10.3 KB
references/backtesting-task.mdmarkdown10.0 KB
references/buyer-language-research.mdmarkdown8.6 KB
references/capacity-evidence.mdmarkdown4.4 KB
references/dedupe.mdmarkdown1.5 KB
references/jev-feature-extraction.mdmarkdown8.2 KB
references/keyword-catalog.mdmarkdown4.9 KB
references/pitfalls.mdmarkdown1.6 KB
references/proven-signals.mdmarkdown1.1 KB
references/quality-gate.mdmarkdown2.8 KB
references/report-template.mdmarkdown6.2 KB
references/scorecard-creation.mdmarkdown3.3 KB
references/scorecard-training.mdmarkdown6.5 KB
references/scoring-delivery.mdmarkdown8.3 KB
references/scoring-diagnostics.mdmarkdown8.9 KB
references/scoring-pitfalls.mdmarkdown2.8 KB
references/signal-interpretation.mdmarkdown2.3 KB
references/step-7-prospects.mdmarkdown1.6 KB
references/technology-evidence.mdmarkdown9.4 KB
references/testing-and-evaluation.mdmarkdown6.8 KB
scripts/analyze_signals.pyscript16.9 KB
scripts/analyze_signals_v2.pyscript28.3 KB
scripts/build_feature_matrix.pyscript4.7 KB
scripts/collect_sources.pyscript16.1 KB
scripts/dedupe_utils.pyscript15.6 KB
scripts/dedupe_utils_v2.pyscript15.7 KB
scripts/find_contacts_v2.pyscript1.9 KB
scripts/hubspot_accounts.pyscript7.1 KB
scripts/jev_features.pyscript7.1 KB
scripts/replay-score.play.tsscript2.3 KB
scripts/requirements-scorecard.txtmarkdown263 B
scripts/scorecard/__init__.pyscript514 B
scripts/scorecard/config.pyscript5.0 KB
scripts/scorecard/explainer_artifacts.pyscript8.4 KB
scripts/scorecard/pipeline.pyscript38.9 KB
scripts/scorecard/report.pyscript30.4 KB
scripts/scorecard/scorer.pyscript17.3 KB
scripts/scorecard/specs.pyscript1.3 KB
scripts/scoring_contract.tsscript7.5 KB
scripts/technology_evidence.pyscript3.8 KB
scripts/tests/contract_check.tsscript1.9 KB
scripts/tests/test_jev_features.pyscript3.4 KB
scripts/tests/test_scorecard.pyscript9.5 KB
scripts/train_scorecard.pyscript20.0 KB
skill-metadata.jsonconfig3.8 KB

references/scoring-pitfalls.md

View on GitHub
# Prevent overfitting and leakage

## Define the decision

Cold-account targeting, engaged-deal forecasting and customer-value estimation have different admissible features and targets. Set the scoring date BEFORE collection. Before close is not before scoring.

Activity counts, champion fields, visits, notes, demo requests and provider installations can be downstream of engagement or purchase. They may be legitimate later-stage features only when available at that later decision and evaluated for that target. Never transfer their apparent lift into cold ICP scoring.

Public content is time-varying too. A current re-scrape cannot retrospectively validate historical prediction. Preserve publication, provider first-seen, retrieval and known-at timestamps separately. Bulk import timestamps are not event times. Use a conservative latest-known-at for all source material in a validation row.

## Separate discovery from confirmation

- Split by time and buyer/parent before choosing phrases. Resolve parent mappings externally; domain equality is insufficient.
- Mine phrases without outcome labels. If labels inform refinements, disclose discovery fitting.
- Freeze aliases, extraction rules, config hashes, cohort rules and the comparator baseline.
- Keep the validation partition untouched. Repeated peeks or revised configurations consume it.
- Lookalikes never count as won; dual-outcome accounts require a defined acquisition/renewal cohort.
- Evaluate incremental performance, calibration and operational utility on a later cohort. A descriptive script does not do this automatically.
- Correct the full tested hypothesis family and log all searches, including negative findings. BH/BY for one invocation cannot correct undisclosed repeated experiments.
- Account-level Fisher tests are not valid cluster-adjusted evidence for correlated subsidiaries. Use grouped bootstrap/model evaluation in a dedicated analysis.
- No fixed n=3 or n=20 establishes power. Plan around baseline, useful effect, uncertainty and class imbalance.

## Missingness and selection

Analyze source coverage by label, size, region and vintage. Restrict descriptive denominators to observed source records and report exclusions. This does NOT eliminate completeness-selection bias. A more thoroughly worked won account often has more data.

Do not require won companies to have more job listings as a quality check. Conditioning on later funnel stages changes the population and can introduce collider bias.

Lost reasons are diagnostic, not automatic targeting rules. Unresponsive can mean poor contact data, timing, message, channel, execution or fit. Verify the correct CRM property and interview evidence before changing ICP.

Never manufacture weights from a raw ratio. Confirm predictive usefulness separately from causality; neither lexical matching nor observational lift establishes causal effect.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall