opensite-ai/opensite-skills / aws-ai-ml
aws-ai-ml
VerifiedSelects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
references/finetuning/references/rlvr_reward_function.md
View on GitHub# RLVR Lambda Reward Function Guide
## What Is a Lambda Reward Function?
For RLVR (Reinforcement Learning from Verifiable Rewards) training, a Lambda reward function is an AWS Lambda that evaluates model outputs during training and returns numerical reward scores. SageMaker invokes this Lambda within the training loop to provide learning signals that guide model optimization.
---
## Helping Users Create Custom Reward Functions
Tell user: I will now review your use case and data, as well as my own resources, to propose a reward function that you can use to train your model. I will do my best to match it to your needs, but I strongly suggest that you review it carefully before starting the training.
### Step 1: Analyze the Use Case
Reward functions are specific to the use case and dataset. Consider the task and data format to understand what constitutes a good output and how to measure it.
1. Review these materials:
- `use_case_spec.md` — problem description and success criteria
- Conversation context — the user's goals
- 20 rows of training data — structure and content of the expected responses
2. Answer these questions internally (involve the user if you need clarification):
- Given the analysis in (1), what makes a good response? A bad response? A partially correct response?
- Which aspects of the response can be verified programmatically?
- Are there specific constraints or formats the output must follow?
- How would the base model's initial responses during early training likely look?
---
### Step 2: Analyze the Structure of the Response
- Which parts of the response contain the content you want to verify programmatically?
- How are those parts delimited? How can they be parsed?
- How rigid should the extraction patterns be, given the 20 rows of data reviewed?
- Are there special formats to account for (fractions, LaTeX, Unicode, Markdown, etc.)? How do they affect the extraction logic?
- Does the base model include a thinking block in its output?
- Does the use case require changing the model's behavior within the thinking block, or only in the final response?
- If warranted, how can the response format/schema be verified programmatically?
- If there is a ground truth in the data:
- Does the model's response need to match it exactly?
- Does a partial match count? If so, how?
- How can you deterministically decide whether the response is close enough to the ground truth?
---
### Step 3: Plan the Verification Logic
- Write a function that extracts the verifiable parts identified in Steps 1 and 2 from the response.
- Identify the most suitable and performant tools for checking format or schema (e.g., which Python libraries?).
- If you need to validate generated code, write a function that executes it and returns a pass/fail/test result with a corresponding reward score.
- Are there keywords to check for? Which ones, and how many need to be present?
- What is the appropriate similarity function for comparing the response to the ground truth?
- If the response contains a block of text where the choice of words can vary slightly and still be correct, how can you verify that it is similar enough to the ground truth?
- Share the plan with the user and get confirmation before proceeding.
### Step 4: Add Anti-Gaming Checks
Add at least two mechanisms to detect and penalize gaming behavior. Common gaming patterns include:
- **Padding** — inserting filler characters to inflate response length
- **Skipping steps** — jumping to a final answer without showing required reasoning
- **Repetition** — filling length requirements with repeated whitespace or words
- **Dummy content** — using placeholder text instead of genuine answers
- **Echo attack** — repeating the prompt or question back as the answer
- **Nonsense** — producing incoherent or irrelevant text
---
### Step 5: Design the Aggregation Method
If the use case allows, think of rewards as a pyramid where each layer depends on the one beneath it. No credit is given for higher layers until lower ones are fully satisfied.
- **Layer 1 (Foundation) — Structure**
- Is the output formatted correctly and machine-parsable?
- Example: If JSON is expected, is it valid JSON? Are all required fields filled?
- **Layer 2 (Core) — Semantics**
- Is the output factually correct and does it deliver real value?
- Example: Can generated code pass unit tests? Is the math answer correct?
- **Layer 3 (Polish) — Behavior**
- Does the output meet operational and safety requirements?
- Example: Is the response concise? Free from toxic content? Complete?
- **Aggregation**
- What is the most suitable weighted distribution across these layers and their sub-components?
- Ensure each component function returns a spread of scores even for low-quality responses. If a component returns 0 for 90%+ of plausible early-training outputs, it will flatten the reward signal and stall learning.
- Briefly share the reasoning with the user and get confirmation.
---
### Step 6: Write the Function as a Python Script
1. Create a file called `lambda_function.py` in the project's scripts directory.
2. Read the `directory-management` reference to determine the correct directory for storing scripts.
3. Consult the reward function templates for structural reference:
- Nova 2.0 Lite → `templates/nova_rlvr_reward_function_source_template.py`
- All other models → `templates/rlvr_reward_function_source_template.py`
**Critical rules:**
- The `lambda_handler` function must be copied to `lambda_function.py` exactly as given in the template. Do not change its signature or internal logic.
- The chat template used in the example reward functions is correct. Use it to extract the assistant's response. Then apply the parsing logic from Step 2 to extract the parts of the response you want to score.
- Do not copy anything beyond the `lambda_handler` and the assistant-response extraction. The rest of the template is an example that **will not work** out of the box. You **must customize** the reward logic based on the use case and data, as described in Steps 1–5. Copying the template's reward logic without customization will likely produce flat rewards, wasting the user's time and compute budget.
**Code writing principles:**
1. **Provide a learning gradient:** Return diverse scores across [-1.0, 1.0], with partial credit for partial answers where appropriate — not just {-1, 0, 1}.
2. **Verify correctly:** Use actual parsing tools (`json.loads`, `ast.parse`, etc.), not string matching.
3. **Include all necessary imports:** Add every required import statement at the top of the file.
4. **Execute fast:** Complete in <100 ms with no API calls or blocking operations.
5. **Be deterministic:** Same input → same output, always.
6. **Be bounded:** The final score must always fall within [-1.0, 1.0]. Add `return min(1.0, max(-1.0, score))` at the end.
7. **Comment thoroughly:** Include detailed comments explaining the reward logic.
### Step 7: Test Locally
Test the reward function by executing it against crafted sample data:
1. **Build test input.** Infer the expected Lambda event and response format from the `lambda_handler` function in the appropriate source template. Choose one prompt from the training data reviewed in Step 1. Construct four test events that mimic what SageMaker sends to the Lambda:
- An **excellent** response — use the response from the data.
- A **partially correct** response — generate one that gets some things right but misses others.
- A **bad** response — generate one that is clearly wrong or off-topic, but without gaming.
- A **gaming** response — generate one that tries to get rewards by gaming.
2. Explain what you are doing and show the user the four responses that you want to test.
3. Write the batch to a temp file (e.g., `/tmp/test_reward_input.json`).
4. **Run the function** Invoke `lambda_function.py` via the shell:
```bash
python3 -c "
import sys, json
sys.path.insert(0, '<project-dir>/scripts')
from lambda_function import lambda_handler
with open('/tmp/test_reward_input.json') as f:
event = json.load(f)
result = lambda_handler(event, None)
print(json.dumps(result, indent=2))
"
```
5. **Verify the output.** Check that:
- All scores fall within [-1.0, 1.0].
- The excellent response scores highest, the bad response scores lowest.
- No errors or exceptions occurred.
6. **Show the user** the test inputs, expected score ordering, and actual scores.
7. If the test fails or scores don't match expectations, fix the function and re-run until it passes. Inform the user about what you are doing.
### Step 8: Check In with the User
- Share the path to the reward function with the user.
- Remind user that this is only a suggestion, and emphasize the need to review the reward function before launching the training. It is up to them to decide if they want to use it, edit it, or choose not to use it.
- Let the user know that the source templates are also available for them under finetuning/templates, if they want to compare your function to them or customize them on their own.
### Step 9: Register the Reward Function in the Finetuning Output
After the reward function is written and tested, generate the registration code that corresponds to **Cell 3** in `code_templates/rlvr.py`. Add the registration code to the finetuning output as Cell 3, following the format already chosen for this session (notebook or script).
Set `reward_function_path` to the path where `lambda_function.py` was saved in Step 6.
```python
from sagemaker.ai_registry.evaluator import Evaluator
# Insert path to lambda_function.py from Step 6 here:
reward_function_path = ""
evaluator = Evaluator.create(
name="[GENERATE A NAME FOR THE EVALUATOR HERE]",
type="RewardFunction",
source=reward_function_path,
)
CUSTOM_REWARD_FUNCTION = evaluator.arn
print(f"Reward Function ARN: {CUSTOM_REWARD_FUNCTION}")
```
Generate an appropriate name for the Evaluator based on the use case and current context.
- Format: lowercase, alphanumeric with hyphens only, 1–20 characters
- Pattern: `[a-zA-Z0-9](-*[a-zA-Z0-9]){0,20}`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |