secondsky/claude-skills / plugins/recommendation-engine/skills/recommendation-engine
recommendation-engine
VerifiedBuild recommendation systems with collaborative filtering, matrix factorization, hybrid approaches. Use for product recommendations, personalization, or encountering cold start, sparsity, quality evaluation issues.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.1 KB |
| references/cold-start-strategies.md | markdown | 19.0 KB |
| references/collaborative-filtering-deep-dive.md | markdown | 15.4 KB |
| references/evaluation-metrics-implementation.md | markdown | 17.0 KB |
| references/matrix-factorization-methods.md | markdown | 17.3 KB |
references/evaluation-metrics-implementation.md
View on GitHub# Evaluation Metrics Implementation
Complete implementations of Precision@K, Recall@K, NDCG, coverage metrics, and statistical significance testing for recommendation systems.
## Overview
Recommendation systems require specialized metrics beyond traditional ML metrics because:
- We care about top-K results, not all predictions
- Ranking quality matters (order of recommendations)
- Diversity and coverage are important
- Online A/B tests measure real business impact
## Precision@K and Recall@K
```python
from typing import List, Set, Dict
import numpy as np
from collections import defaultdict
import logging
logger = logging.getLogger(__name__)
def precision_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
"""
Precision@K: Fraction of recommended items in top-K that are relevant.
Args:
recommended: List of recommended item IDs (ordered by score)
relevant: Set of truly relevant item IDs
k: Number of top recommendations to consider
Returns:
Precision@K score in [0, 1]
Example:
>>> recommended = [1, 3, 5, 7, 9]
>>> relevant = {1, 5, 10}
>>> precision_at_k(recommended, relevant, k=5)
0.4 # 2 out of 5 recommendations are relevant
"""
if k <= 0:
return 0.0
top_k = recommended[:k]
relevant_in_top_k = len([item for item in top_k if item in relevant])
return relevant_in_top_k / k
def recall_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
"""
Recall@K: Fraction of relevant items that appear in top-K.
Args:
recommended: List of recommended item IDs
relevant: Set of truly relevant item IDs
k: Number of top recommendations
Returns:
Recall@K score in [0, 1]
Example:
>>> recommended = [1, 3, 5, 7, 9]
>>> relevant = {1, 5, 10}
>>> recall_at_k(recommended, relevant, k=5)
0.667 # 2 out of 3 relevant items retrieved
"""
if len(relevant) == 0:
return 0.0
top_k = recommended[:k]
relevant_in_top_k = len([item for item in top_k if item in relevant])
return relevant_in_top_k / len(relevant)
def f1_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
"""
F1@K: Harmonic mean of Precision@K and Recall@K.
Returns:
F1 score in [0, 1]
"""
prec = precision_at_k(recommended, relevant, k)
rec = recall_at_k(recommended, relevant, k)
if prec + rec == 0:
return 0.0
return 2 * (prec * rec) / (prec + rec)
def average_precision_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
"""
Average Precision@K (AP@K): Rewards putting relevant items earlier.
Computes precision at each relevant item position, then averages.
Args:
recommended: Ordered list of recommendations
relevant: Set of relevant items
k: Maximum number of recommendations to consider
Returns:
AP@K score in [0, 1]
Example:
>>> recommended = [1, 2, 3, 4, 5]
>>> relevant = {1, 3, 5}
>>> average_precision_at_k(recommended, relevant, k=5)
0.756 # (1/1 + 2/3 + 3/5) / 3
"""
if len(relevant) == 0:
return 0.0
top_k = recommended[:k]
score = 0.0
num_hits = 0.0
for i, item in enumerate(top_k):
if item in relevant:
num_hits += 1.0
score += num_hits / (i + 1.0)
return score / min(len(relevant), k)
def mean_average_precision_at_k(
recommendations: Dict[int, List[int]],
ground_truth: Dict[int, Set[int]],
k: int
) -> float:
"""
Mean Average Precision@K (MAP@K): Average AP@K across all users.
Args:
recommendations: Dict mapping user_id -> list of recommended items
ground_truth: Dict mapping user_id -> set of relevant items
k: Number of recommendations
Returns:
MAP@K score in [0, 1]
"""
ap_scores = []
for user_id in recommendations:
if user_id in ground_truth:
ap = average_precision_at_k(
recommendations[user_id],
ground_truth[user_id],
k
)
ap_scores.append(ap)
return np.mean(ap_scores) if ap_scores else 0.0
```
## Normalized Discounted Cumulative Gain (NDCG)
NDCG measures ranking quality with position-based discounting.
```python
def dcg_at_k(recommended: List[int], relevance_scores: Dict[int, float], k: int) -> float:
"""
Discounted Cumulative Gain@K.
DCG = sum(rel_i / log2(i + 1)) for i in [1, k]
Args:
recommended: Ordered list of recommended items
relevance_scores: Dict mapping item_id -> relevance score
k: Number of top items
Returns:
DCG score
"""
dcg = 0.0
for i, item_id in enumerate(recommended[:k]):
rel = relevance_scores.get(item_id, 0.0)
# Position discount: log2(i + 2) because i starts at 0
dcg += rel / np.log2(i + 2)
return dcg
def ndcg_at_k(
recommended: List[int],
relevance_scores: Dict[int, float],
k: int
) -> float:
"""
Normalized Discounted Cumulative Gain@K.
NDCG = DCG / IDCG
Where IDCG is DCG of the ideal ranking (sorted by relevance).
Args:
recommended: Ordered list of recommended items
relevance_scores: Dict mapping item_id -> relevance (higher = more relevant)
k: Number of top items
Returns:
NDCG score in [0, 1]
Example:
>>> recommended = [3, 1, 5, 2, 4]
>>> relevance = {1: 3, 2: 2, 3: 3, 4: 1, 5: 2}
>>> ndcg_at_k(recommended, relevance, k=5)
0.952 # Close to ideal ranking
"""
# Compute DCG for actual recommendations
dcg = dcg_at_k(recommended, relevance_scores, k)
# Compute IDCG (ideal DCG) by sorting items by relevance
ideal_ranking = sorted(
relevance_scores.items(),
key=lambda x: x[1],
reverse=True
)
ideal_items = [item_id for item_id, _ in ideal_ranking]
idcg = dcg_at_k(ideal_items, relevance_scores, k)
if idcg == 0:
return 0.0
return dcg / idcg
def mean_ndcg_at_k(
recommendations: Dict[int, List[int]],
relevance_scores: Dict[int, Dict[int, float]],
k: int
) -> float:
"""
Mean NDCG@K across all users.
Args:
recommendations: Dict[user_id -> list of recommended items]
relevance_scores: Dict[user_id -> Dict[item_id -> relevance]]
k: Number of recommendations
Returns:
Mean NDCG@K
"""
ndcg_scores = []
for user_id, recs in recommendations.items():
if user_id in relevance_scores:
ndcg = ndcg_at_k(recs, relevance_scores[user_id], k)
ndcg_scores.append(ndcg)
return np.mean(ndcg_scores) if ndcg_scores else 0.0
```
## Coverage and Diversity Metrics
```python
def catalog_coverage(
recommendations: Dict[int, List[int]],
catalog_size: int,
k: int
) -> float:
"""
Catalog Coverage: Fraction of catalog items that appear in recommendations.
Measures diversity across all users.
Args:
recommendations: Dict[user_id -> list of recommendations]
catalog_size: Total number of items in catalog
k: Number of recommendations per user
Returns:
Coverage in [0, 1]
Example:
>>> recs = {
... 0: [1, 2, 3],
... 1: [2, 3, 4],
... 2: [3, 4, 5]
... }
>>> catalog_coverage(recs, catalog_size=100, k=3)
0.05 # 5 unique items out of 100
"""
recommended_items = set()
for user_recs in recommendations.values():
recommended_items.update(user_recs[:k])
return len(recommended_items) / catalog_size
def gini_coefficient(recommendations: Dict[int, List[int]], k: int) -> float:
"""
Gini Coefficient: Measure of recommendation concentration.
0 = perfectly equal distribution (all items recommended equally)
1 = perfect inequality (only one item recommended to everyone)
Lower is better (more diverse).
Args:
recommendations: Dict[user_id -> list of recommendations]
k: Number of recommendations per user
Returns:
Gini coefficient in [0, 1]
"""
# Count how many times each item was recommended
item_counts = defaultdict(int)
for user_recs in recommendations.values():
for item in user_recs[:k]:
item_counts[item] += 1
# Sort counts
counts = sorted(item_counts.values())
n = len(counts)
if n == 0:
return 0.0
# Compute Gini
# G = (2 * sum(i * x_i)) / (n * sum(x_i)) - (n + 1) / n
cumsum = np.cumsum(counts)
gini = (2 * np.sum((np.arange(1, n + 1) * counts))) / (n * cumsum[-1]) - (n + 1) / n
return gini
def intra_list_diversity(recommended: List[int], item_features: np.ndarray) -> float:
"""
Intra-List Diversity: Average dissimilarity between recommended items.
Measures how diverse a single recommendation list is.
Args:
recommended: List of recommended item IDs
item_features: Array (n_items, n_features) of item feature vectors
Returns:
Average pairwise distance in [0, 1]
"""
from sklearn.metrics.pairwise import cosine_distances
if len(recommended) < 2:
return 0.0
# Get feature vectors for recommended items
rec_features = item_features[recommended]
# Compute pairwise distances
distances = cosine_distances(rec_features)
# Average distance (excluding diagonal)
n = len(recommended)
total_distance = np.sum(distances) - np.trace(distances) # Exclude diagonal
avg_distance = total_distance / (n * (n - 1))
return avg_distance
```
## Cross-Validation for Recommenders
```python
from sklearn.model_selection import KFold
def time_based_split(
interactions: np.ndarray,
timestamp_col: int,
test_ratio: float = 0.2
):
"""
Split based on time (realistic for recommendations).
Train on past interactions, test on future.
Args:
interactions: Array with columns [user_id, item_id, timestamp, ...]
timestamp_col: Column index containing timestamps
test_ratio: Fraction of data for test set
Returns:
train_interactions, test_interactions
"""
# Sort by timestamp
sorted_interactions = interactions[interactions[:, timestamp_col].argsort()]
# Split at time threshold
split_idx = int(len(sorted_interactions) * (1 - test_ratio))
train = sorted_interactions[:split_idx]
test = sorted_interactions[split_idx:]
logger.info(f"Time-based split: {len(train)} train, {len(test)} test")
return train, test
def leave_one_out_cv(user_item_matrix: np.ndarray):
"""
Leave-one-out cross-validation for each user.
For each user, hold out one interaction for testing.
Args:
user_item_matrix: Matrix (n_users, n_items)
Yields:
(train_matrix, test_matrix) pairs
"""
n_users, n_items = user_item_matrix.shape
for user_id in range(n_users):
# Get user's interactions
user_interactions = user_item_matrix[user_id].nonzero()[0]
if len(user_interactions) < 2:
continue # Need at least 2 interactions
# Randomly select one to hold out
test_item = np.random.choice(user_interactions)
# Create train and test matrices
train_matrix = user_item_matrix.copy()
train_matrix[user_id, test_item] = 0
test_matrix = np.zeros_like(user_item_matrix)
test_matrix[user_id, test_item] = user_item_matrix[user_id, test_item]
yield train_matrix, test_matrix
```
## Statistical Significance Testing
```python
from scipy import stats
def paired_t_test(
metrics_a: List[float],
metrics_b: List[float],
alpha: float = 0.05
) -> Dict:
"""
Paired t-test to compare two recommendation models.
Tests if difference in metrics is statistically significant.
Args:
metrics_a: Metric values (e.g., NDCG@10) for model A, one per user
metrics_b: Metric values for model B, same users
alpha: Significance level
Returns:
Dict with test results
Example:
>>> ndcg_baseline = [0.7, 0.8, 0.75, 0.82, 0.79]
>>> ndcg_new_model = [0.72, 0.83, 0.76, 0.85, 0.81]
>>> result = paired_t_test(ndcg_baseline, ndcg_new_model)
>>> print(result['significant']) # True if new model is better
"""
if len(metrics_a) != len(metrics_b):
raise ValueError("Metric lists must have same length")
# Compute differences
differences = np.array(metrics_b) - np.array(metrics_a)
# Perform paired t-test
t_statistic, p_value = stats.ttest_rel(metrics_a, metrics_b)
# Effect size (Cohen's d)
mean_diff = np.mean(differences)
std_diff = np.std(differences, ddof=1)
cohens_d = mean_diff / std_diff if std_diff > 0 else 0.0
return {
'mean_improvement': mean_diff,
'p_value': p_value,
'significant': p_value < alpha,
't_statistic': t_statistic,
'cohens_d': cohens_d,
'interpretation': 'significant' if p_value < alpha else 'not significant'
}
def bootstrap_confidence_interval(
metric_values: List[float],
n_bootstrap: int = 1000,
confidence: float = 0.95
) -> tuple:
"""
Compute confidence interval for a metric using bootstrap.
Args:
metric_values: List of metric values
n_bootstrap: Number of bootstrap samples
confidence: Confidence level (e.g., 0.95 for 95% CI)
Returns:
(lower_bound, upper_bound)
"""
bootstrap_means = []
for _ in range(n_bootstrap):
# Resample with replacement
sample = np.random.choice(metric_values, size=len(metric_values), replace=True)
bootstrap_means.append(np.mean(sample))
# Compute percentiles
alpha = 1 - confidence
lower = np.percentile(bootstrap_means, alpha / 2 * 100)
upper = np.percentile(bootstrap_means, (1 - alpha / 2) * 100)
return lower, upper
```
## Complete Evaluation Pipeline
```python
class RecommenderEvaluator:
"""
Comprehensive evaluation of recommendation models.
"""
def __init__(self, k_values: List[int] = [5, 10, 20]):
"""
Args:
k_values: List of K values to evaluate at
"""
self.k_values = k_values
def evaluate(
self,
model,
test_data: Dict[int, Set[int]],
catalog_size: int
) -> Dict:
"""
Evaluate model on test data.
Args:
model: Trained recommender model with recommend() method
test_data: Dict[user_id -> set of relevant items]
catalog_size: Total number of items
Returns:
Dict of metrics
"""
results = defaultdict(dict)
# Generate recommendations for all users
recommendations = {}
for user_id in test_data:
try:
recs = model.recommend(user_id, n=max(self.k_values))
recommendations[user_id] = [item for item, score in recs]
except Exception as e:
logger.warning(f"Failed to generate recs for user {user_id}: {e}")
continue
# Compute metrics at each K
for k in self.k_values:
# Precision, Recall, F1
precisions = []
recalls = []
for user_id, relevant_items in test_data.items():
if user_id in recommendations:
prec = precision_at_k(recommendations[user_id], relevant_items, k)
rec = recall_at_k(recommendations[user_id], relevant_items, k)
precisions.append(prec)
recalls.append(rec)
results[f'precision@{k}'] = np.mean(precisions) if precisions else 0.0
results[f'recall@{k}'] = np.mean(recalls) if recalls else 0.0
results[f'f1@{k}'] = (
2 * results[f'precision@{k}'] * results[f'recall@{k}']
/ (results[f'precision@{k}'] + results[f'recall@{k}'])
if (results[f'precision@{k}'] + results[f'recall@{k}']) > 0 else 0.0
)
# Coverage
results[f'coverage@{k}'] = catalog_coverage(recommendations, catalog_size, k)
# Diversity (Gini)
results[f'gini@{k}'] = gini_coefficient(recommendations, k)
logger.info("Evaluation complete:")
for metric, value in results.items():
logger.info(f" {metric}: {value:.4f}")
return dict(results)
```
## Example Usage
```python
# Evaluate model
evaluator = RecommenderEvaluator(k_values=[5, 10, 20])
# Test data: dict of user_id -> set of relevant items
test_data = {
0: {10, 25, 42},
1: {5, 17, 33},
# ... more users
}
# Run evaluation
metrics = evaluator.evaluate(
model=my_recommender,
test_data=test_data,
catalog_size=1000
)
# Compare two models statistically
baseline_ndcg = [...] # NDCG@10 for each test user with baseline
new_model_ndcg = [...] # NDCG@10 for each test user with new model
test_result = paired_t_test(baseline_ndcg, new_model_ndcg)
if test_result['significant']:
print(f"New model is significantly better (p={test_result['p_value']:.4f})")
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |