opensite-ai/opensite-skills / aws-ai-ml
aws-ai-ml
VerifiedSelects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
references/model-evaluation/references/llmaaj-builtin-evaluation.md
View on GitHub# LLM-as-Judge with Built-in Metrics: Alignment Guide
This file describes the process for aligning on built-in metrics to use for model evaluations with LLMaaJ.
## Select Metrics
Refer to the metrics tables below for the full list of metrics with descriptions and common combinations.
Based on the user's task and data, recommend specific metrics with reasoning:
> "Based on your [task], I recommend these metrics:
>
> - [metric1]: [why it matters for this task]
>
> Does this look good, or do you want to consider other metrics?"
⏸ **Wait for user to confirm.**
Tips:
- Start with the common combinations from the metrics file as a baseline
- Adjust based on what you know about the user's task and data
- If the user pushes back, understand why and adjust — don't just agree
## LLM-as-Judge Built-in Metrics
SageMaker provides 11 built-in metrics for LLM-as-Judge evaluation, organized into Quality and Responsible AI categories.
## Quality Metrics
| Metric | Description | When to Use |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| Correctness | Measures if the model's response to the prompt is correct. If a reference response (ground truth) is provided in the dataset, the evaluator considers this when scoring. | QA, math problems, factual tasks |
| Completeness | Measures how well the model's response answers every question in the prompt. If a reference response is provided, the evaluator considers this when scoring. | Multi-part questions, comprehensive answers, summarization |
| Faithfulness | Identifies whether the response contains information not found in the prompt to measure how faithful the response is to the available context. | RAG applications, context-grounded responses |
| Helpfulness | Measures how helpful the model's response is using factors including whether it follows instructions, is sensible and coherent, and anticipates implicit needs. | General assistance, customer service, broad evaluation |
| Coherence | Measures how coherent the response is by identifying logical gaps, inconsistencies, and contradictions. | Long-form content, reasoning tasks, explanations |
| Relevance | Measures how relevant the answer is to the prompt. | All tasks - commonly used baseline metric |
| FollowingInstructions | Measures how well the model's response respects the exact directions found in the prompt. | Instruction-following tasks, structured outputs, specific formatting |
| ProfessionalStyleAndTone | Measures how appropriate the response's style, formatting, and tone is for a professional setting. | Business communications, formal writing |
## Responsible AI Metrics
| Metric | Description | When to Use |
| ------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| Harmfulness | Evaluates whether the response contains harmful content. | Safety evaluation, content moderation |
| Stereotyping | Evaluates whether content in the response contains stereotypes of any kind (either positive or negative). | Fairness evaluation, bias detection |
| Refusal | Determines if the response directly declines to answer the prompt or rejects the request by providing reasons. | Safety evaluation, understanding model boundaries |
## Usage in Code
In code, these metrics are specified as `Builtin.Correctness`, `Builtin.Completeness`, etc. When discussing with users, use natural language names.
## Common Metric Combinations
- **QA/Math tasks** → Correctness, Completeness, Faithfulness, Relevance
- **Summarization** → Completeness, Coherence, Relevance
- **General assistance** → Helpfulness, Relevance, FollowingInstructions
- **Safety evaluation** → Harmfulness, Stereotyping, Refusal
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |