Skip to content

hollandkevint/data-product-operator / skills/data-product-validation

data-product-validation

Verified

Score whether a data product idea is worth building before committing resources. Validation scorecard, experiment design, and go/kill decisions. Use when evaluating feasibility, making go/no-go decisions, validating demand, sizing bets, or when someone asks "is this worth building?" or "should we invest in this?"

Validation Scorecard

Score each dimension 1-5. Multiply all five for a composite score (max 3,125).

1. Demand Frequency

How often do consumers need this answer?

ScoreFrequencyExample
5Daily or real-timeICU readmission risk scores
4WeeklyRegional performance reports
3MonthlyQuarterly business reviews
2QuarterlyAnnual planning data
1Rarely or onceAd-hoc executive request

2. Decision Impact

What happens when consumers don't have this data?

ScoreImpactExample
5Critical decision blockedCan't discharge patients without risk assessment
4Significant delay or costTeam spends 20+ hours/week on manual workaround
3Moderate inconvenienceReport takes 3 hours instead of 5 minutes
2Minor frictionSlightly slower process, acceptable workaround exists
1Mild inconvenienceNice to have, nobody changes behavior without it

3. Workaround Effort

What are consumers doing today instead?

ScoreEffortExample
5Custom tooling builtAnalyst maintains a 47-tab Excel model updated daily
4Significant manual process3 people spend 2 days/week compiling reports
3Partial automationScript exists but breaks frequently, needs babysitting
2Simple workaroundQuick Excel export, takes 30 minutes
1Haven't triedNobody has attempted to solve this yet

4. Data Feasibility

Can we actually build this with available data?

ScoreFeasibilityExample
5Data exists, quality verifiedSource audited, passes data-quality-assessment checks
4Data exists, quality unknownSource available but no quality baseline established
3Data exists, known issuesSource has gaps or accuracy problems that need fixing first
2Data partially existsNeed to combine 3+ sources, some missing
1Data doesn't existWould need new data collection or acquisition

Cross-reference data-quality-assessment for the 5-dimension scoring. A Data Feasibility score of 1 kills the idea regardless of other scores.

5. Schema Risk

How locked-in are consumers once we ship?

ScoreRiskExample
5Easily re-modeledInternal API, few consumers, versioning supported
4Moderate couplingDashboard consumed by 5-10 users, change is disruptive but manageable
3Significant coupling20+ downstream queries depend on current schema
2High couplingExternal consumers or contractual SLAs on schema shape
1Breaking change impossibleRegulated output, schema changes require compliance review

Thresholds

Composite ScoreDecisionAction
250+BuildShape a pitch, allocate a cycle
100-249InvestigateRun a 1-week experiment to de-risk the lowest-scoring dimension
<100KillDocument why and move on. Revisit only if new evidence surfaces

A Data Feasibility score of 1 kills the idea at any composite score. You can't build a data product without data.

Experiment Types

When the scorecard says "Investigate," pick the cheapest experiment that addresses the weakest dimension:

Sample Query (1 day): Write the SQL. Run it against production data. Does the result answer the consumer's question? Tests Data Feasibility and reveals quality issues fast. If the query returns garbage, kill early.

Manual Pipeline (1 week): Build the transformation by hand for one consumer. Deliver a spreadsheet or flat file. Measure: did they use it? Did it change a decision? Tests Demand Frequency and Decision Impact with real behavior.

Schema Prototype (2-3 weeks): Build a minimal star schema with one fact table and 2-3 dimensions. Load a month of data. Let 2-3 consumers query it. Tests Schema Risk and reveals modeling assumptions before they calcify.

Type 1 vs Type 2 Decisions

Not all data product decisions are equal:

Type 1 (irreversible): Schema design decisions. Data source commitments. Terminology mappings published to external consumers. SCD strategies once historical data accumulates. These deserve full validation and the scorecard process.

Type 2 (reversible): Adding a metric to an existing dashboard. New filters on an existing report. Feature additions to established products. These don't need a scorecard. Just build and measure.

Discovery budget should match decision type. Spend 2 weeks validating a Type 1. Spend 2 hours on a Type 2.

brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 321kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 197kUpdated Claude CodeCodexVerifiedInstall