foryourhealth111-pixel/Vibe-Skills / bundled/skills/evaluating-machine-learning-models
evaluating-machine-learning-models
VerifiedEvaluate trained machine learning models with the right metrics and comparison logic. Use for benchmark review, threshold selection, calibration, validation, and model comparison; not for feature engineering or leakage auditing.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 1.5 KB |
| assets/README.md | markdown | 359 B |
| assets/visualization_script.py | script | 5.5 KB |
| references/README.md | markdown | 397 B |
| scripts/README.md | markdown | 407 B |
| scripts/data_loader.py | script | 2.8 KB |
| scripts/evaluate_model.py | script | 2.8 KB |
| scripts/metrics_calculator.py | script | 2.8 KB |
scripts/metrics_calculator.py
View on GitHub#!/usr/bin/env python3
"""
model-evaluation-suite - metrics_calculator.py
Script to calculate evaluation metrics based on model predictions and ground truth.
Generated: 2025-12-10 03:48:17
"""
import os
import sys
import json
import argparse
from pathlib import Path
from datetime import datetime
def process_file(file_path: Path) -> bool:
"""Process individual file."""
if not file_path.exists():
print(f"β File not found: {file_path}")
return False
print(f"π Processing: {file_path}")
# Add processing logic here based on skill requirements
# This is a template that can be customized
try:
if file_path.suffix == '.json':
with open(file_path) as f:
data = json.load(f)
print(f" β Valid JSON with {len(data)} keys")
else:
size = file_path.stat().st_size
print(f" β File size: {size:,} bytes")
return True
except Exception as e:
print(f" β Error: {e}")
return False
def process_directory(dir_path: Path) -> int:
"""Process all files in directory."""
processed = 0
failed = 0
for file_path in dir_path.rglob('*'):
if file_path.is_file():
if process_file(file_path):
processed += 1
else:
failed += 1
return processed, failed
def main():
parser = argparse.ArgumentParser(
description="Script to calculate evaluation metrics based on model predictions and ground truth."
)
parser.add_argument('input', help='Input file or directory')
parser.add_argument('--output', '-o', help='Output directory')
parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
parser.add_argument('--config', '-c', help='Configuration file')
args = parser.parse_args()
input_path = Path(args.input)
print(f"π model-evaluation-suite - metrics_calculator.py")
print(f" Category: ai-ml")
print(f" Plugin: model-evaluation-suite")
print(f" Input: {input_path}")
if args.config:
if Path(args.config).exists():
with open(args.config) as f:
config = json.load(f)
print(f" Config: {args.config}")
# Process input
if input_path.is_file():
success = process_file(input_path)
result = 0 if success else 1
elif input_path.is_dir():
processed, failed = process_directory(input_path)
print(f"\nπ SUMMARY")
print(f" β
Processed: {processed}")
print(f" β Failed: {failed}")
result = 0 if failed == 0 else 1
else:
print(f"β Invalid input: {input_path}")
result = 1
if result == 0:
print("\nβ
Completed successfully")
else:
print("\nβ Completed with errors")
return result
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |