Skip to content

getaero-io/gtm-eng-skills / skills/deepline-scoring

deepline-scoring

Verified

Use when discovering niche signals, auditing ICP or won/lost evidence, rescoring accounts, or building account and lead scoring Plays. Triggers on fit scoring, engagement scoring, backtesting, holdout evaluation, predictive lift, external proxies, and scoring leakage. Skip pure outreach copy or contributor skill installation tasks.

FileKindSize
.gitignoreother26 B
SKILL.mdskill md12.4 KB
package.jsonmanifest114 B
references/account-website-features.mdmarkdown10.3 KB
references/backtesting-task.mdmarkdown10.0 KB
references/buyer-language-research.mdmarkdown8.6 KB
references/capacity-evidence.mdmarkdown4.4 KB
references/dedupe.mdmarkdown1.5 KB
references/jev-feature-extraction.mdmarkdown8.2 KB
references/keyword-catalog.mdmarkdown4.9 KB
references/pitfalls.mdmarkdown1.6 KB
references/proven-signals.mdmarkdown1.1 KB
references/quality-gate.mdmarkdown2.8 KB
references/report-template.mdmarkdown6.2 KB
references/scorecard-creation.mdmarkdown3.3 KB
references/scorecard-training.mdmarkdown6.5 KB
references/scoring-delivery.mdmarkdown8.3 KB
references/scoring-diagnostics.mdmarkdown8.9 KB
references/scoring-pitfalls.mdmarkdown2.8 KB
references/signal-interpretation.mdmarkdown2.3 KB
references/step-7-prospects.mdmarkdown1.6 KB
references/technology-evidence.mdmarkdown9.4 KB
references/testing-and-evaluation.mdmarkdown6.8 KB
scripts/analyze_signals.pyscript16.9 KB
scripts/analyze_signals_v2.pyscript28.3 KB
scripts/build_feature_matrix.pyscript4.7 KB
scripts/collect_sources.pyscript16.1 KB
scripts/dedupe_utils.pyscript15.6 KB
scripts/dedupe_utils_v2.pyscript15.7 KB
scripts/find_contacts_v2.pyscript1.9 KB
scripts/hubspot_accounts.pyscript7.1 KB
scripts/jev_features.pyscript7.1 KB
scripts/replay-score.play.tsscript2.3 KB
scripts/requirements-scorecard.txtmarkdown263 B
scripts/scorecard/__init__.pyscript514 B
scripts/scorecard/config.pyscript5.0 KB
scripts/scorecard/explainer_artifacts.pyscript8.4 KB
scripts/scorecard/pipeline.pyscript38.9 KB
scripts/scorecard/report.pyscript30.4 KB
scripts/scorecard/scorer.pyscript17.3 KB
scripts/scorecard/specs.pyscript1.3 KB
scripts/scoring_contract.tsscript7.5 KB
scripts/technology_evidence.pyscript3.8 KB
scripts/tests/contract_check.tsscript1.9 KB
scripts/tests/test_jev_features.pyscript3.4 KB
scripts/tests/test_scorecard.pyscript9.5 KB
scripts/train_scorecard.pyscript20.0 KB
skill-metadata.jsonconfig3.8 KB

references/scoring-delivery.md

View on GitHub
# Scoring delivery contract

## Freeze the decision

Record customer/workspace, analysis unit, supported identifier types, full scoring population, target/horizon, cutoff policy, requested dimensions, reference population, destination and spend scope. Resolve “main” to a named table/project and version before writes. Never switch customer data through another customer's workspace.

Return every input row with its original key. Invalid identity, insufficient evidence, and out-of-scope rows remain visible with null score/grade and a reason. Never silently drop them or assign D for missing data.

Score the full requested universe, not a post-opportunity filter. Evaluation can include a settled acquisition won/lost subset and a separately named engaged subset. Report each denominator. For win-within-H, require complete follow-up for negatives; pending rows are censored. Preserve renewal/expansion and dual-outcome episode rules. Accounts, deals, contacts and parent groups are distinct grains.

## Four dimensions, requested outputs only

| Dimension          | Inputs                                                                                                                                | Excluded                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| account_fit        | Website workflow/pain and business-model concepts, plus external trade, geography, size, footprint, verified technology and ownership | Visits, replies, deal stage, AE notes, internal pain/budget |
| account_engagement | Dated relevant first-party activity, deduplicated across contacts                                                                     | Fit points; lifetime counters with no recency               |
| lead_fit           | Account fit plus current role, seniority, responsibility and persona match                                                            | Engagement points; seller-assigned champion labels          |
| lead_engagement    | Dated individual activity, relevance, channel and recency                                                                             | Account size and title points                               |

Keep hiring spikes/acquisitions in separately named timing features unless the customer defines their place. Distinguish steady department structure from a fresh hiring event. Separation does not imply statistical independence. A customer-requested combined priority score is a separate tested policy; preserve the components.

## Required website feature layer

For account-fit and propensity deliveries, follow [account website features](account-website-features.md): collect the relevant source pages, export complete account/concept observations, review their meaning, and join their approved encodings into the scorer. Include billing/revenue/customer-model concepts. Show per-feature contributions or exclusion reasons. A summary of keyword prevalence or a few sampled quotes does not establish that an account was scored on them. An unchanged approved model can be replayed for parity; label missing website inputs and evaluate the proposed layer separately. Do not replace it or invent positive weights without acceptance evidence.

## Point-in-time and proxies

Every observation carries entity ID/scope, feature/value/units, source ID/URL, source class, event_at, known_at, retrieved_at, extraction version and collection status. Default historical account cutoff is before the earliest relevant outreach/interaction or opportunity creation, according to the customer's decision. Verify histories; creation/import date may not be the original event. Enforce strict earlier-than for a pre-contact/pre-opportunity cutoff.

Do not declare a feature safe because it is firmographic or mostly pre-close. A fact learned later requires audited historical availability and a distinct reconstructed-backtest vintage. Current observations cannot validate older outcomes. Retain historical snapshots.

AE facts are discovery targets: map `internal variable → hypothesis → external proxy → source/units → match rules → availability → proxy error → incremental predictive test`. CSR count might map to dated named staff or department estimates. Review velocity/service footprint are call-demand proxies, NOT measured calls. Evaluate count estimates against CRM actuals separately from win prediction. Never assume proxy equivalence.

## A/B/C/D grades

Default bands, highest scores first: A top 10%; B next 15% (10–25%); C next 25% (25–50%); D remaining 50%. These are unequal percentile bands, not quartiles, probabilities, or validation criteria.

Freeze a representative reference distribution per dimension/model version. A singleton or new batch uses that reference, never itself. Persist reference ID, definition, date, N, model hash, sorted scores and tie policy. Do not derive cutoffs on the final holdout when measuring generalization.

Default tie policy: percentile = 100 × (strictly lower reference values + half the equal reference values) / N. A ≥90; B ≥75 and <90; C ≥50 and <75; D <50. Equal raw scores receive equal grades. Report actual shares and tie mass instead of claiming exact quotas. Fewer than two distinct reference values yields `insufficient_reference_variation`. Exact-capacity queues may use a disclosed stable-ID tie-break for queue position only. Missing scores stay unscored. Reference changes are versioned migrations.

## Required runnable artifact

Search/describe fitting Plays first. Inputs may be domain, company LinkedIn, email, person LinkedIn, or name plus company context; implement requested forms rather than promising every form. Ambiguous names fail with candidates/reason. Verify company/person URL types and parent/branch scope.

Create a reusable end-to-end Play with a shared pure scorer. Use the same transforms, missing-value policy, learned coefficients or approved heuristic rules, and reference distribution offline and online. No per-row LLM-generated weights. Prevalence ratios are not model coefficients. Pin provider schemas and extraction versions; retain provider attempts and source lineage. Separate paid enrichment from deterministic scoring so changing grades does not repurchase data.

Output: input, resolved identity, enriched attributes, requested numeric scores/grades/percentiles, model/reference versions, evidence, eligibility decisions, confidence/coverage, status and miss_reason. Export every row. Preserve run IDs, final export, observed costs and provider statuses. A domain-to-cached-row replay is NOT live enrichment for unseen domains; label it accordingly.

## Backtest and acceptance

- Assert no future/AE-derived inputs, outcome-derived features, or parent split overlap.
- Verify expected IDs, row counts, duplicates, outcome conflicts and pagination across all raw populations.
- Apply collection uniformly; report coverage-only negative controls.
- Fit selection, imputation, transforms and tuning inside training folds. Freeze temporal evaluation once; repeated peeks consume it.
- Compare trade/geo/size baseline and feature-family ablations. Report grade N/wins/losses/pending, precision@K, lift@K, PR-AUC and uncertainty at natural prevalence. Band shares are not performance.
- Calibrate before claiming probabilities. Test outreach impact prospectively, not from observational association alone.
- Fixtures: normal, sparse, ambiguous, wrong entity, stale/future source, import/backfill, negative evidence, provider error, ties, exact boundaries, zero variation, unknown domain, reference mismatch, singleton/batch parity, replay after version changes.
- Invariants: engagement cannot change fit; AE edits cannot change pre-contact scores; batch membership cannot change frozen-reference grades; customer data cannot cross workspaces.

Completion states: `research_only`, `replay_only`, `exploratory_end_to_end`, `validated_for_named_use_case`. Name unmet gates. Preserve report/HTML, signal inventory, evidence, score export, model/reference artifacts, Play source/checks, backtest, costs and unresolved rows.

Finish with the implementation-test stage routed from the skill. Known-answer regression, selected win/loss sanity checks and untouched predictive validation are separate gates; preserve independent oracles and failure receipts for each.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall