Skip to content

opensite-ai/opensite-skills / aws-ai-ml

aws-ai-ml

Verified

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md8.1 KB
references/dataset-evaluation/overview.mdmarkdown5.4 KB
references/dataset-evaluation/references/custom-scorer-evaluation-dataset-formats.mdmarkdown3.6 KB
references/dataset-evaluation/references/strategy_data_requirements.mdmarkdown4.8 KB
references/dataset-evaluation/scripts/format_detector.pyscript26.7 KB
references/dataset-transformation/code_templates/transformation.pyscript1.5 KB
references/dataset-transformation/overview.mdmarkdown13.3 KB
references/dataset-transformation/references/code_output_guide.mdmarkdown2.8 KB
references/dataset-transformation/references/dataset_transformation_code.mdmarkdown5.2 KB
references/dataset-transformation/references/sagemaker_dataset_formats.mdmarkdown2.9 KB
references/dataset-transformation/scripts/transformation_tools.pyscript4.8 KB
references/directory-management/overview.mdmarkdown1.9 KB
references/endpoint-diagnostics/overview.mdmarkdown3.5 KB
references/endpoint-diagnostics/references/minimum_iam_policy.mdmarkdown1.2 KB
references/endpoint-diagnostics/scripts/collect_diagnostics.pyscript12.8 KB
references/finetuning-technique/overview.mdmarkdown2.1 KB
references/finetuning-technique/references/finetune_technique_selection_guide.mdmarkdown2.3 KB
references/finetuning-technique/scripts/get_recipes.pyscript817 B
references/finetuning/code_templates/dpo.pyscript6.4 KB
references/finetuning/code_templates/rlaif_builtin.pyscript5.8 KB
references/finetuning/code_templates/rlaif_custom_prompt.pyscript6.3 KB
references/finetuning/code_templates/rlvr.pyscript6.5 KB
references/finetuning/code_templates/sft.pyscript6.1 KB
references/finetuning/overview.mdmarkdown11.4 KB
references/finetuning/references/code_output_guide.mdmarkdown3.2 KB
references/finetuning/references/continuous_customization.mdmarkdown8.4 KB
references/finetuning/references/eula_links.mdmarkdown7.3 KB
references/finetuning/references/rlaif_guide.mdmarkdown3.8 KB
references/finetuning/references/rlvr_reward_function.mdmarkdown10.0 KB
references/finetuning/scripts/mlflow_reference.pyscript841 B
references/finetuning/templates/nova_rlvr_reward_function_source_template.pyscript12.8 KB
references/finetuning/templates/rlvr_reward_function_source_template.pyscript9.1 KB
references/manage-mlflow/overview.mdmarkdown4.8 KB
references/manage-mlflow/references/app-creation-workflow.mdmarkdown3.2 KB
references/manage-mlflow/references/app-deletion-workflow.mdmarkdown2.3 KB
references/manage-mlflow/references/app-discovery-workflow.mdmarkdown1.5 KB
references/manage-mlflow/references/app-update-workflow.mdmarkdown1.7 KB
references/manage-mlflow/references/sm-mlflow-guide.mdmarkdown1.3 KB
references/model-deployment/code_templates/deploy-jumpstart-sagemaker.pyscript7.5 KB
references/model-deployment/code_templates/deploy-nova-bedrock.pyscript2.0 KB
references/model-deployment/code_templates/deploy-nova-sagemaker.pyscript1.9 KB
references/model-deployment/code_templates/deploy-oss-bedrock.pyscript4.0 KB
references/model-deployment/code_templates/deploy-oss-sagemaker.pyscript2.6 KB
references/model-deployment/code_templates/optimize-benchmark.pyscript4.6 KB
references/model-deployment/code_templates/optimize-huggingface-to-s3.pyscript2.0 KB
references/model-deployment/code_templates/optimize-recommendation.pyscript6.4 KB
references/model-deployment/overview.mdmarkdown16.6 KB
references/model-deployment/references/code_output_guide.mdmarkdown3.2 KB
references/model-deployment/references/deploy-jumpstart-sagemaker.mdmarkdown14.3 KB
references/model-deployment/references/deploy-nova-bedrock.mdmarkdown4.4 KB
references/model-deployment/references/deploy-nova-sagemaker.mdmarkdown5.8 KB
references/model-deployment/references/deploy-oss-bedrock.mdmarkdown5.2 KB
references/model-deployment/references/deploy-oss-sagemaker.mdmarkdown12.9 KB
references/model-deployment/references/model-licenses.mdmarkdown11.9 KB
references/model-deployment/references/optimization/benchmark-workflow.mdmarkdown12.7 KB
references/model-deployment/references/optimization/benchmarking-guidance.mdmarkdown8.9 KB
references/model-deployment/references/optimization/huggingface-to-s3.mdmarkdown3.1 KB
references/model-deployment/references/optimization/interpreting-results.mdmarkdown8.5 KB
references/model-deployment/references/optimization/recommendation-deploy.mdmarkdown1.5 KB
references/model-deployment/references/optimization/recommendation-options.mdmarkdown3.1 KB
references/model-deployment/references/optimization/recommendation-workflow.mdmarkdown10.3 KB
references/model-evaluation/code_templates/custom_scorer_evaluator.pyscript2.5 KB
references/model-evaluation/code_templates/llmaaj_evaluator.pyscript2.8 KB
references/model-evaluation/overview.mdmarkdown5.2 KB
references/model-evaluation/references/code_output_guide.mdmarkdown3.3 KB
references/model-evaluation/references/create-reward-function.mdmarkdown3.0 KB
references/model-evaluation/references/custom-lambda-scorer.mdmarkdown4.4 KB
references/model-evaluation/references/custom-scorer-evaluation.mdmarkdown10.5 KB
references/model-evaluation/references/evaluation-type-guide.mdmarkdown10.1 KB
references/model-evaluation/references/llmaaj-builtin-evaluation.mdmarkdown5.0 KB
references/model-evaluation/references/llmaaj-custom-evaluation.mdmarkdown2.5 KB
references/model-evaluation/references/llmaaj-evaluation.mdmarkdown14.6 KB
references/model-evaluation/references/supported-judge-models.mdmarkdown2.9 KB
references/model-evaluation/scripts/nova_reward_function_source_template.pyscript13.1 KB
references/model-evaluation/scripts/reward_function_source_template.pyscript9.2 KB
references/model-evaluation/scripts/validate_custom_metrics.pyscript3.5 KB
references/model-selection/overview.mdmarkdown10.3 KB
references/model-selection/references/benchmarks/agentic-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/coding-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/gpqa.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/hle.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/ifbench.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/intelligence-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/mmmu-pro.mdmarkdown3.0 KB
references/model-selection/references/benchmarks/tau2.mdmarkdown2.8 KB
references/model-selection/references/model-licenses.mdmarkdown7.3 KB
references/model-selection/references/model-selection.mdmarkdown8.2 KB
references/model-selection/references/select-for-deployment.mdmarkdown19.8 KB
references/model-selection/scripts/filter_deployable_models.pyscript10.1 KB
references/model-selection/scripts/get_deployable_models.pyscript21.7 KB
references/model-selection/scripts/get_model_names.pyscript1.4 KB
references/model-selection/scripts/search_deployable_models.pyscript34.9 KB
references/planning/overview.mdmarkdown10.3 KB
references/planning/references/deploy-base-model-plan.mdmarkdown699 B
references/planning/references/evaluate-first-plan.mdmarkdown1.8 KB
references/planning/references/input-output-contracts.mdmarkdown3.1 KB
references/planning/references/model-customization-plan.mdmarkdown1.5 KB
references/planning/references/region-availability.mdmarkdown1.2 KB
references/planning/references/skill-routing-constraints.mdmarkdown2.7 KB
references/processing-jobs/code_templates/dataset_definition_processing.pyscript4.1 KB
references/processing-jobs/code_templates/framework_processing.pyscript3.0 KB
references/processing-jobs/code_templates/script_processing.pyscript2.9 KB
references/processing-jobs/code_templates/spark_processing.pyscript3.7 KB
references/processing-jobs/overview.mdmarkdown6.7 KB
references/processing-jobs/references/configuration.mdmarkdown4.6 KB
references/processing-jobs/references/troubleshooting.mdmarkdown2.4 KB
references/sdk-getting-started/overview.mdmarkdown643 B
references/sdk-getting-started/references/execution-role-setup.mdmarkdown2.4 KB
references/sdk-getting-started/references/sagemaker-python-sdk-setup.mdmarkdown2.1 KB
references/training-jobs/code_templates/custom_training.pyscript6.2 KB
references/training-jobs/code_templates/jumpstart_training.pyscript5.6 KB
references/training-jobs/code_templates/recipe_training.pyscript5.7 KB
references/training-jobs/overview.mdmarkdown9.1 KB
references/training-jobs/references/configuration.mdmarkdown4.0 KB
references/training-jobs/references/troubleshooting.mdmarkdown4.4 KB
references/use-case-specification/overview.mdmarkdown2.0 KB
references/use-case-specification/references/spec-for-deployment.mdmarkdown3.9 KB
references/use-case-specification/references/spec-for-finetuning.mdmarkdown2.0 KB

references/model-evaluation/references/custom-scorer-evaluation.md

View on GitHub
# Custom Scorer Evaluation

Guide the user through the process for evaluating a model with Custom Scorer (built-in Prime Math/Code or custom Lambda).

## Workflow

### Step 0: Consider prior context

Before proceeding, silently think about the context you have about the user's project, including conversation history and file reads. You should use that knowledge, and avoid asking questions you already know the answer to.

### Step 1: Validate Custom Scorer compatibility

Before proceeding, confirm one thing:

1. **Does the user have an evaluation dataset?**

If the check fails (the user has no eval dataset), tell the user and offer to help them pick an alternative:

> "Custom Scorer evaluation requires an evaluation dataset. Would you like help choosing a different evaluation type?"

If they want help choosing a different evaluation type → break this workflow and read `references/evaluation-type-guide.md`.

If the check passes, proceed.

### Step 2: Understand the task

For this step, you need: to understand **the task the model is trained to do.**
If you know this already, skip this step. If not, ask the user:

> "What task is this model trained to do?"

### Step 3: Get evaluation dataset

For this step, you need: **the evaluation dataset S3 path.**
If you know this already, skip this step. If not, ask the user:

> "Where's your evaluation dataset stored in S3?"

### Step 4: Choose scorer type

For this step, you need to know **which scorer to use.**

**If the model is Nova:** built-in scorers (Prime Math, Prime Code) are not supported for Nova models. Inform the user and proceed with Custom Lambda:

> "For Nova models, only Custom Lambda scoring is supported. Built-in scorers (Prime Math, Prime Code) won't produce results. Let's set up a Custom Lambda scorer."

Then read `references/custom-lambda-scorer.md` and follow its instructions. Return here and proceed to Step 5.

**If the model is OSS:** ask if you don't already know from context:

> "Which type of scorer would you like to use?
>
> 1. **Prime Math** — built-in scorer for math problems (checks answer correctness)
> 2. **Prime Code** — built-in scorer for coding problems (executes code against test cases)
> 3. **Custom Lambda** — your own scoring logic as a Lambda function. You can use an existing registered evaluator or create a new one.
>
> Which would you prefer?"

- If built-in (Prime Math or Prime Code) → note the choice and proceed to Step 5.
- If custom Lambda → read `references/custom-lambda-scorer.md` and follow its instructions to resolve the evaluator. Then return here and proceed to Step 5. You MUST follow these instructions before moving on.

### Step 5: Validate dataset format

IMPORTANT: you MUST validate that dataset, to ensure that it is the correct format. Note that there are precise requirements based on model and evaluation type, so you cannot skip this step.

Reference the **dataset-evaluation** reference to perform this validation.

### Step 6: Determine evaluation scope

For this step, you need to know **which model(s) to evaluate.**

If you already know from context, confirm and move on. Otherwise, ask:

> "Would you like to evaluate:
>
> 1. **Just your fine-tuned model**
> 2. **Just a base model**
> 3. **Both, with a comparison**

⏸ Wait for user approval.

### Step 7: Resolve Model Package ARN

**This step only applies if the evaluation scope includes the fine-tuned model (option 1 or 3 from Step 6).** If the user chose base model only, skip to Step 8.

For this step, you need: **the Model Package ARN of the fine-tuned model.**

If you already have it from prior context, confirm with the user and move on. Otherwise, ask:

> "What's the Model Package ARN (or group name) of your fine-tuned model?"

If they provide a group name, resolve the ARN by running `aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1`. Use the latest version's `ModelPackageArn`.

**Validate the resolved ARN:**

- Must look like: `arn:aws:sagemaker:REGION:ACCOUNT:model-package/NAME/VERSION`
- If it's a group ARN (`:model-package-group/`), resolve to a package ARN by running `aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1`. Use the latest version's `ModelPackageArn`.
- If it contains `:model-package/` but does NOT end with a version number (e.g., `/1`), resolve it: run `aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1`.
- If it contains `/DataSet/`, `/TrainingJob/`, or other non-model-package resource types, flag it: "That looks like a [Dataset/TrainingJob] ARN, not a model package ARN. Could you double-check?"
- Verify it exists by running `aws sagemaker describe-model-package --model-package-name <arn>`. If this fails, tell the user the ARN wasn't found and ask them to double-check.

### Step 8: Resolve base model

**This step only applies if the evaluation scope includes the base model (option 2 or 3 from Step 6).** If the user chose fine-tuned only, skip to Step 9.

For **comparison mode** (option 3): the base model is resolved automatically from the fine-tuned model's lineage — no additional input needed.

For **base model only** (option 2): you need a JumpStart model ID (e.g., `meta-textgeneration-llama-3-2-1b-instruct`). Check if you already know it from context. If not, ask:

> "What's the JumpStart model ID of the base model you'd like to evaluate?"

<!-- TODO: Add guidance for helping the user find their JumpStart model ID. -->

### Step 9: Resolve IAM role ARN

For this step, you need: **an IAM role ARN with permissions for evaluation.**

If you already know the IAM role ARN from conversation context, confirm with the user and move on.

If not, but you know the training job name, look up the role from that training job, since it should be a suitable role:

- Run `aws sagemaker describe-training-job --training-job-name <name>` and extract `RoleArn`.
- Confirm with the user.

If you don't have either, ask:

> "What IAM role ARN should I use for the evaluation job?"

**Important:** If the user chose a custom Lambda scorer, the IAM role must also have `lambda:InvokeFunction` permission for the scorer's Lambda function. Mention this to the user:

> "Note: since you're using a custom Lambda scorer, this role needs `lambda:InvokeFunction` permission for your scorer function. Does your role have that?"

### Step 10: Resolve AWS Region

For this step, you need: **the AWS region for the evaluation job.**
If you already know it from context, confirm and move on. Otherwise, ask.

### Step 11: Resolve S3 output path

For this step, you need: **an S3 path for evaluation output.**
If you already know it from context, confirm and move on. Otherwise, ask.

### Step 12: MLflow experiment name (optional)

For this step, you need to know: **the MLflow experiment name.**
If you already know this from context, confirm and move on. If not, the name should follow this pattern:
`<project-name>-eval`.

Perform this step quietly. The user does not need to know about this. This is to make sure their results are viewable alongside other experimental work they're doing.

### Step 13: Final Checks

IMPORTANT: Before moving on, you MUST think through every step of this workflow. Did you complete it, or did you have a good reason for skipping? If not, please complete it now. In particular, make sure you have carefully validated the dataset and the lambda against the correct requirements.

### Step 14: Confirm configuration

Summarize everything and ask for approval:

> "Here's the evaluation setup:
>
> - Task: [task]
> - Dataset: [path]
> - Scorer: [Prime Math / Prime Code / Custom Lambda ARN]
> - Model: [Model Package ARN or JumpStart model ID]
> - Evaluation scope: [fine-tuned only / base only / both with comparison]
> - IAM role: [ARN]
> - Region: [region]
> - S3 output: [path]
> - MLflow experiment name: [MLflow experiment name]
>
> Does this look right?"

⏸ Wait for user approval.

### Step 15: Generate code

Read `../references/code_output_guide.md` for output format rules.

If no project directory exists, load the **directory-management** reference to set one up.

Read `code_templates/custom_scorer_evaluator.py`, substitute the collected values into the placeholders, and write the cells. The template uses `# Cell N: Label` markers — each marker starts a new notebook cell, with everything between one marker and the next becoming that cell's content.

### Step 16: Post-generation

**Notebook mode:**

```
To run:
1. Cell 1 — configuration and SDK install
2. Cell 2 — start evaluation
3. Cell 3 — polls status automatically (~25-60 min)
4. Cell 4 — show results

```

**Script mode:**

Evaluation can take hours depending on your dataset. Present the user with options:

> "Would you like me to:
>
> 1. Leave it to you — run with `python scripts/[script_name]`
> 2. Run it and wait until it's done
> 3. Start it but don't wait — we can check status later"

- **Option 1:** Done. Wait for user to come back.
- **Option 2:** Execute the script as-is. `execution.wait()` polls until complete. Report results.
- **Option 3:** Remove the `execution.wait()` call, execute, report the evaluation ARN.

Note: `evaluate()` does not accept a `wait` parameter. It always returns immediately. Blocking is done via `execution.wait(target_status="Succeeded")`.

**Checking status:**

- `describe-pipeline-execution --pipeline-execution-arn ARN` → `PipelineExecutionStatus`
- `list-pipeline-execution-steps --pipeline-execution-arn ARN` → per-step `StepStatus`, `FailureReason`

**Showing results after completion:**

- Run: `EvaluationPipelineExecution.get(arn=ARN).show_results()`

## FAQ

**Q: What metrics do I get with Custom Scorer?**
A: Both built-in and custom scorers automatically produce standard NLP metrics (F1, ROUGE, BLEU) alongside your custom scores.

**Q: Does my IAM role need special permissions for Custom Scorer?**
A: Yes — if using a custom Lambda scorer, the IAM role needs `lambda:InvokeFunction` permission for the scorer's Lambda function. Built-in scorers (Prime Math/Code) don't require additional permissions.

**Q: Can I create a new reward function through this skill?**
A: Yes — if you choose Custom Lambda and don't have an existing evaluator, the agent will walk you through creating one from a template and registering it via `Evaluator.create`.

## Nova Model Notes

Custom Scorer evaluation works with Nova models via Custom Lambda. Built-in scorers (Prime Math, Prime Code) are not supported — the pipeline will run without error, but the scorer will not execute.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall