opensite-ai/opensite-skills / aws-ai-ml
aws-ai-ml
VerifiedSelects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
references/finetuning/references/rlaif_guide.md
View on GitHub# RLAIF Fine-Tuning Guide
RLAIF (Reinforcement Learning from AI Feedback) uses a Bedrock LLM as a judge to score model outputs during training. No human-labeled preference pairs are needed — the judge evaluates responses in real time.
## How RLAIF Differs from RLVR
- **RLVR**: reward comes from a Lambda function (verifiable, deterministic)
- **RLAIF**: reward comes from a Bedrock LLM judge (flexible, open-ended)
- **Best for**: summarization, helpfulness, instruction-following, open-ended quality
## Reward Model Options
The `reward_model_id` sets the Bedrock LLM used as judge. To get the current list of available models, run:
```bash
venv/bin/python3 -c "from sagemaker.train.constants import _ALLOWED_REWARD_MODEL_IDS; import json; print(json.dumps(_ALLOWED_REWARD_MODEL_IDS, indent=2))"
```
Present the output to the user as a numbered list showing each model name and its available regions, then ask them to pick one.
---
## Option 1: Builtin Reward Prompt
The simplest path. Choose one of the four builtin prompts — the SDK maps it to the corresponding Jinja template in the Hub recipe.
Pass the builtin name directly as the `reward_prompt` parameter:
- `"Builtin.Summarize"` — evaluates summarization quality
- `"Builtin.Faithfulness"` — evaluates factual consistency with source
- `"Builtin.ChainOfThought"` — evaluates step-by-step reasoning quality
- `"Builtin.Evaluation"` — general response quality evaluation
**When to use**: When one of the four builtin prompts matches the use case well enough. Ask the user which one fits, or suggest based on the task.
**Under the hood**: `reward_prompt="Builtin.Summarize"` sets the hyperparameter `judge_prompt_template` to the matching template. No `Evaluator.create()` call needed.
See `code_templates/rlaif_builtin.py` for the full code template.
---
## Option 2: Custom Reward Prompt
When the builtin prompts don't fit the use case, register a custom Jinja prompt file as a `RewardPrompt` evaluator.
Suitable for: domain-specific quality, structured output validation, or multi-criteria scoring.
**Key difference from RLVR**:
- RLVR uses `Evaluator.create(type=REWARD_FUNCTION)` → deploys a Lambda function
- RLAIF uses `Evaluator.create(type=REWARD_PROMPT)` → uploads a text/Jinja file to S3
The Bedrock judge receives the prompt and evaluates the model output. No Lambda is involved.
### Steps
1. **Write the prompt file** — create a `.jinja` file with a suitable name in the project's scripts directory. The prompt should instruct the judge how to evaluate the model's response. It can reference `{{ prompt }}` and `{{ response }}` template variables.
**To help user write the prompt** - think:
- What should the judge look for in a good response?
- What should it penalize?
- Should it return a score, a label, or a ranking?
2. **Register the prompt as an evaluator**:
```python
from sagemaker.ai_registry.evaluator import Evaluator
from sagemaker.ai_registry.air_constants import REWARD_PROMPT
reward_prompt_evaluator = Evaluator.create(
name="[GENERATE A NAME HERE]", # lowercase alphanumeric + hyphens, max 20 chars
type=REWARD_PROMPT,
source="path/to/custom_reward_prompt.jinja", # local file path or S3 URI
sagemaker_session=sagemaker_session,
wait=True
)
REWARD_PROMPT_ARN = reward_prompt_evaluator.arn
print(f"Evaluator ARN: {REWARD_PROMPT_ARN}")
```
3. **Pass the ARN as `reward_prompt`** to `RLAIFTrainer` (instead of the builtin string).
See `code_templates/rlaif_custom_prompt.py` for the full code template.
---
## Notes
- `mlflow_experiment_name` and `mlflow_run_name` are optional but recommended for tracking.
- For continued fine-tuning from a previously trained model, pass a `ModelPackage` object as `model` instead of a base model string. See `continuous_customization.md`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |