DAAF-Contribution-Community/daaf / .claude/skills/data-scientist
data-scientist
VerifiedData science methodology and method-selection routing for quantitative research. Covers EDA, data validation, descriptive analysis, causal inference (IV, DiD, RD, synthetic control), clustering/PCA/UMAP, supervised ML, geospatial analysis, network analysis, and visualization design. Contains the canonical method-to-library routing tree, routed by execution language — Python: statsmodels (OLS/GLM/time series), pyfixest (FE/IV/DiD), linearmodels (RE/GMM/SUR), svy (complex surveys), scikit-learn (clustering/prediction ML), geopandas (spatial), igraph (network analysis); R: r-stats (OLS/GLM/time series), fixest (FE/IV/DiD), plm (panel RE/between, dynamic GMM), survey-r (complex surveys), tidymodels (ML), sf-terra (spatial), igraph-r (network analysis). Load the routed tool-specific skill before giving tool-specific advice or writing code — library skills encode environment constraints and curated caveats absent from general knowledge.
references/data-documentation.md
View on GitHub# Data Documentation
Understanding existing documentation and creating documentation when none exists.
## Contents
- [Questions to Ask About Data](#questions-to-ask-about-data)
- [Understanding Columns](#understanding-columns)
- [Data Provenance](#data-provenance)
- [Working with Undocumented Data](#working-with-undocumented-data)
- [Creating Documentation](#creating-documentation)
- [Red Flags to Surface](#red-flags-to-surface)
## Questions to Ask About Data
Before any analysis, seek answers to these questions.
### About Data Source
| Question | Why It Matters |
|----------|----------------|
| Where does this data come from? | Determines reliability, update frequency |
| What system/process generated it? | Understanding generation can reveal biases |
| Who owns or maintains the source? | Knowing who to ask for clarification |
| When was it last updated? | Staleness affects relevance |
| How is data extracted/exported? | Export process might introduce issues |
### About Data Collection
| Question | Why It Matters |
|----------|----------------|
| What was the collection method? | Survey, sensor, transaction, manual entry each have biases |
| What time period does it cover? | Affects generalizability |
| Are there known gaps in coverage? | Missing data might not be random |
| What population does it represent? | Sampling bias affects conclusions |
| Was any filtering applied before export? | May be missing important subsets |
### About Data Quality
| Question | Why It Matters |
|----------|----------------|
| Are there known quality issues? | Avoid rediscovering known problems |
| How are errors handled? | Might be silently dropped or flagged |
| What validation exists at entry? | Determines trust in values |
| Has data been cleaned already? | Avoid double-cleaning |
| Are there special codes for missing/invalid? | -999, "N/A", etc. need handling |
### About Business Context
| Question | Why It Matters |
|----------|----------------|
| What business process generates this data? | Context helps interpret values |
| What decisions are made using it? | Understand stake and scrutiny level |
| Who are the domain experts? | Know who to consult |
| Are there regulatory requirements? | May constrain what you can do |
| What would be surprising in this data? | Domain knowledge catches errors |
## Understanding Columns
For each column, understand these attributes:
### Essential Attributes
| Attribute | Description | Example |
|-----------|-------------|---------|
| **Name** | Column identifier | `customer_id` |
| **Description** | What it represents in business terms | "Unique identifier assigned when customer account created" |
| **Data type** | Technical type | Integer, String, Datetime |
| **Nullability** | Can it be missing? | "Yes, for anonymous visitors" |
| **Domain/Valid values** | Allowed values | "Positive integers", "USA/CAN/MEX" |
### Extended Attributes (When Available)
| Attribute | Description |
|-----------|-------------|
| **Source** | Where the value originates |
| **Calculation logic** | If derived, how it's computed |
| **Update frequency** | How often it changes |
| **Dependencies** | Related columns or external data |
| **Known issues** | Documented problems or caveats |
### Questions for Each Column
When documentation is sparse, ask:
1. **What does this column represent in the real world?**
2. **What unit is it measured in?** (dollars, cents, kilograms, count)
3. **What does a null value mean?** (missing, not applicable, unknown)
4. **What are the valid values?** (range, enumeration, format)
5. **Is it directly captured or derived?** (if derived, from what?)
6. **Can it change over time?** (historical vs. current state)
## Data Provenance
Understanding where data comes from and how it got here.
### Lineage Questions
```
Data Flow:
Source System → Extraction → Transformation → Storage → Export → Your Analysis
↓ ↓ ↓ ↓ ↓
What system? How often? What changes? What format? Any filters?
```
### What to Document
1. **Source system(s)**: Original database, API, file system
2. **Extraction method**: SQL query, API call, file transfer
3. **Transformations applied**: Joins, filters, aggregations, cleaning
4. **Storage location**: Data warehouse, data lake, local file
5. **Export process**: How you received the data
### Why Provenance Matters
| Provenance Issue | Risk |
|-----------------|------|
| Unknown source | Can't validate correctness |
| Undocumented transforms | May have hidden assumptions |
| Multiple sources joined | Possible key mismatches |
| Old extraction | Data may be stale |
| Filtered export | Missing important subsets |
## Working with Undocumented Data
When you receive data with no documentation, follow this process.
### Step 1: Generate Basic Profile
**Python:**
```python
import polars as pl
# Generate documentation from data inspection: one row per column
profile_rows = []
for col in df.columns:
profile_rows.append({
"column": col,
"dtype": str(df[col].dtype),
"null_count": df[col].null_count(),
"null_pct": round(df[col].null_count() / len(df) * 100, 1),
"unique_count": df[col].n_unique(),
# Joined to a string, as in the R version: columns differ in type, so a
# list column of raw values would not share one dtype across rows
"sample_values": ", ".join(
str(v) for v in df[col].drop_nulls().unique(maintain_order=True).head(5).to_list()
),
})
profile = pl.DataFrame(profile_rows)
print(profile)
```
**R:**
```r
library(dplyr)
library(tibble)
# Generate documentation from data inspection
profile <- tibble(
column = names(df),
dtype = sapply(df, \(x) paste(class(x), collapse = "/")),
null_count = sapply(df, \(x) sum(is.na(x))),
null_pct = round(sapply(df, \(x) sum(is.na(x))) / nrow(df) * 100, 1),
unique_count = sapply(df, n_distinct),
sample_values = sapply(df, \(x) paste(head(unique(na.omit(x)), 5), collapse = ", "))
)
print(profile)
```
### Step 2: Identify Granularity
**Critical question: What does each row represent?**
**Python:**
```python
# Test candidate keys for uniqueness -- identify what uniquely identifies each row
candidates = []
# Test single columns
for col in df.columns:
if df[col].n_unique() == len(df):
candidates.append([col])
# Test date + entity patterns
date_cols = [c for c in df.columns if "date" in c.lower() or "time" in c.lower()]
# A column matching both patterns (e.g. "update_time_id") stays a date column:
# selecting it twice would give a duplicate output name, which Polars rejects
id_cols = [c for c in df.columns if "id" in c.lower() and c not in date_cols]
for date_col in date_cols:
for id_col in id_cols:
if df.select([date_col, id_col]).n_unique() == len(df):
candidates.append([date_col, id_col])
print("Candidate unique keys:")
for key in candidates:
print(f" {key}")
```
**R:**
```r
# Test candidate keys for uniqueness -- identify what uniquely identifies each row
candidates <- list()
# Test single columns
for (col in names(df)) {
if (n_distinct(df[[col]]) == nrow(df)) {
candidates <- c(candidates, list(col))
}
}
# Test date + entity patterns
date_cols <- names(df)[grepl("date|time", names(df), ignore.case = TRUE)]
id_cols <- names(df)[grepl("id", names(df), ignore.case = TRUE)]
for (date_col in date_cols) {
for (id_col in id_cols) {
unique_count <- df |> distinct(across(all_of(c(date_col, id_col)))) |> nrow()
if (unique_count == nrow(df)) {
candidates <- c(candidates, list(c(date_col, id_col)))
}
}
}
cat("Candidate unique keys:\n")
for (key in candidates) {
cat(" [", paste(key, collapse = ", "), "]\n")
}
```
### Step 3: Infer Column Meanings
**Python:**
```python
# Infer what each column might represent
for col in df.columns:
inferences = []
n_unique = df[col].n_unique()
pct_unique = n_unique / len(df) * 100
sample_vals = df[col].drop_nulls().unique(maintain_order=True).head(5).to_list()
# Name-based inference
name_lower = col.lower()
if "id" in name_lower:
inferences.append("Likely identifier/key")
if "date" in name_lower or "time" in name_lower:
inferences.append("Likely temporal")
if "amt" in name_lower or "amount" in name_lower or "price" in name_lower:
inferences.append("Likely monetary value")
if "count" in name_lower or "num" in name_lower or "qty" in name_lower:
inferences.append("Likely count/quantity")
if "flag" in name_lower or "is_" in name_lower or "has_" in name_lower:
inferences.append("Likely boolean indicator")
# Cardinality-based inference
if pct_unique > 95:
inferences.append("High cardinality (ID? Free text?)")
elif n_unique == 2:
inferences.append("Binary (boolean?)")
elif pct_unique < 1:
inferences.append("Low cardinality (categorical)")
print(f"\n{col} ({df[col].dtype}): {pct_unique:.1f}% unique")
print(" Sample:", ", ".join(str(v) for v in sample_vals))
if inferences:
print(" Inferences:", "; ".join(inferences))
```
**R:**
```r
# Infer what each column might represent
for (col in names(df)) {
inferences <- c()
n_unique <- n_distinct(df[[col]])
pct_unique <- n_unique / nrow(df) * 100
sample_vals <- head(unique(na.omit(df[[col]])), 5)
# Name-based inference
name_lower <- tolower(col)
if (grepl("id", name_lower)) inferences <- c(inferences, "Likely identifier/key")
if (grepl("date|time", name_lower)) inferences <- c(inferences, "Likely temporal")
if (grepl("amt|amount|price", name_lower)) inferences <- c(inferences, "Likely monetary value")
if (grepl("count|num|qty", name_lower)) inferences <- c(inferences, "Likely count/quantity")
if (grepl("flag|is_|has_", name_lower)) inferences <- c(inferences, "Likely boolean indicator")
# Cardinality-based inference
if (pct_unique > 95) inferences <- c(inferences, "High cardinality (ID? Free text?)")
else if (n_unique == 2) inferences <- c(inferences, "Binary (boolean?)")
else if (pct_unique < 1) inferences <- c(inferences, "Low cardinality (categorical)")
cat(sprintf("\n%s (%s): %.1f%% unique\n", col, class(df[[col]]), pct_unique))
cat(" Sample:", paste(sample_vals, collapse = ", "), "\n")
if (length(inferences) > 0) cat(" Inferences:", paste(inferences, collapse = "; "), "\n")
}
```
### Step 4: Document As You Learn
As you discover what columns mean, CREATE documentation:
**Python:**
```python
# Build documentation dictionary as you learn
data_dictionary = {
"customer_id": {
"description": "Unique customer identifier",
"type": "Integer",
"nullable": False,
"notes": "Assigned at account creation, never reused"
},
"order_date": {
"description": "Date order was placed",
"type": "Date",
"nullable": False,
"notes": "In UTC timezone"
},
# Add more as you discover...
}
```
**R:**
```r
# Build documentation as a tibble as you learn
data_dictionary <- tribble(
~column, ~description, ~type, ~nullable, ~notes,
"customer_id", "Unique customer identifier", "integer", FALSE, "Assigned at account creation, never reused",
"order_date", "Date order was placed", "Date", FALSE, "In UTC timezone",
# Add more as you discover...
)
print(data_dictionary)
```
## Creating Documentation
### Minimum Viable Data Dictionary
At minimum, document:
```markdown
## Data Dictionary: [Dataset Name]
### Overview
- **Source**: [Where data comes from]
- **Granularity**: [What each row represents]
- **Date range**: [Time period covered]
- **Row count**: [Number of records]
### Columns
| Column | Type | Description | Nullable | Notes |
|--------|------|-------------|----------|-------|
| customer_id | Int64 | Unique customer identifier | No | Primary key |
| order_date | Date | Date order was placed | No | UTC timezone |
| amount | Float64 | Order total in USD | No | Includes tax |
| status | String | Order status | No | Values: pending, shipped, delivered, cancelled |
### Known Issues
- [List any known data quality issues]
### Change Log
- [Date]: Initial documentation created
```
### Documentation Template
**Python:**
```python
# Generate a documentation template from the DataFrame
dataset_name = "Customer Orders"
template = f"""# Data Dictionary: {dataset_name}
## Overview
- **Source**: [TODO: Document source]
- **Granularity**: [TODO: What does each row represent?]
- **Date range**: [TODO: Time period covered]
- **Row count**: {len(df):,}
- **Column count**: {len(df.columns)}
## Columns
| Column | Type | Nullable | Unique | Description |
|--------|------|----------|--------|-------------|
"""
for col in df.columns:
dtype = df[col].dtype
nullable = "Yes" if df[col].null_count() > 0 else "No"
unique_pct = round(df[col].n_unique() / len(df) * 100, 1)
template += f"| {col} | {dtype} | {nullable} | {unique_pct}% | [TODO] |\n"
template += """
## Known Issues
- [TODO: Document known issues]
## Change Log
- [DATE]: Documentation created
"""
print(template)
```
**R:**
```r
# Generate a documentation template from data frame
dataset_name <- "Customer Orders"
cat(sprintf("# Data Dictionary: %s\n\n", dataset_name))
cat("## Overview\n")
cat("- **Source**: [TODO: Document source]\n")
cat("- **Granularity**: [TODO: What does each row represent?]\n")
cat("- **Date range**: [TODO: Time period covered]\n")
cat(sprintf("- **Row count**: %s\n", format(nrow(df), big.mark = ",")))
cat(sprintf("- **Column count**: %d\n\n", ncol(df)))
cat("## Columns\n\n")
cat("| Column | Type | Nullable | Unique | Description |\n")
cat("|--------|------|----------|--------|-------------|\n")
for (col in names(df)) {
dtype <- class(df[[col]])
nullable <- if (any(is.na(df[[col]]))) "Yes" else "No"
unique_pct <- round(n_distinct(df[[col]]) / nrow(df) * 100, 1)
cat(sprintf("| %s | %s | %s | %.1f%% | [TODO] |\n", col, dtype, nullable, unique_pct))
}
cat("\n## Known Issues\n- [TODO: Document known issues]\n\n")
cat("## Change Log\n- [DATE]: Documentation created\n")
```
## Red Flags to Surface
When reviewing data, flag these issues for discussion:
### Data Quality Red Flags
| Red Flag | Concern |
|----------|---------|
| >10% missing in critical column | May bias analysis |
| Unexpected data types | Possible parsing/export error |
| Values outside reasonable range | Data quality issue or different units |
| Dates in the future | Likely data entry errors |
| Duplicate primary keys | Data integrity violation |
| Inconsistent encodings | Same thing represented differently |
### Documentation Red Flags
| Red Flag | Concern |
|----------|---------|
| No documentation exists | Risk of misinterpretation |
| Documentation is outdated | Schema may have changed |
| Column meanings unclear | May use data incorrectly |
| No provenance information | Can't validate correctness |
| Unknown null conventions | May mishandle missing data |
### Sampling Red Flags
| Red Flag | Concern |
|----------|---------|
| Convenience sample | May not represent population |
| Low response rate | Non-response bias |
| Specific subsets only | Findings may not generalize |
| Time-limited data | Seasonality effects |
| Geographic limitations | Regional bias |
### Report Template for Red Flags
```markdown
## Data Quality Concerns
### Critical Issues
1. [Issue]: [Description and impact]
- Recommendation: [How to address]
### Important Findings
1. [Finding]: [Description]
- Implication: [What it means for analysis]
### Questions for Data Owner
1. [Question about unclear aspect]
2. [Question about data quality issue]
### Assumptions Made
1. [Assumption]: [Why it was necessary]
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |