Skip to content

opensite-ai/opensite-skills / aws-ai-ml

aws-ai-ml

Verified

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md8.1 KB
references/dataset-evaluation/overview.mdmarkdown5.4 KB
references/dataset-evaluation/references/custom-scorer-evaluation-dataset-formats.mdmarkdown3.6 KB
references/dataset-evaluation/references/strategy_data_requirements.mdmarkdown4.8 KB
references/dataset-evaluation/scripts/format_detector.pyscript26.7 KB
references/dataset-transformation/code_templates/transformation.pyscript1.5 KB
references/dataset-transformation/overview.mdmarkdown13.3 KB
references/dataset-transformation/references/code_output_guide.mdmarkdown2.8 KB
references/dataset-transformation/references/dataset_transformation_code.mdmarkdown5.2 KB
references/dataset-transformation/references/sagemaker_dataset_formats.mdmarkdown2.9 KB
references/dataset-transformation/scripts/transformation_tools.pyscript4.8 KB
references/directory-management/overview.mdmarkdown1.9 KB
references/endpoint-diagnostics/overview.mdmarkdown3.5 KB
references/endpoint-diagnostics/references/minimum_iam_policy.mdmarkdown1.2 KB
references/endpoint-diagnostics/scripts/collect_diagnostics.pyscript12.8 KB
references/finetuning-technique/overview.mdmarkdown2.1 KB
references/finetuning-technique/references/finetune_technique_selection_guide.mdmarkdown2.3 KB
references/finetuning-technique/scripts/get_recipes.pyscript817 B
references/finetuning/code_templates/dpo.pyscript6.4 KB
references/finetuning/code_templates/rlaif_builtin.pyscript5.8 KB
references/finetuning/code_templates/rlaif_custom_prompt.pyscript6.3 KB
references/finetuning/code_templates/rlvr.pyscript6.5 KB
references/finetuning/code_templates/sft.pyscript6.1 KB
references/finetuning/overview.mdmarkdown11.4 KB
references/finetuning/references/code_output_guide.mdmarkdown3.2 KB
references/finetuning/references/continuous_customization.mdmarkdown8.4 KB
references/finetuning/references/eula_links.mdmarkdown7.3 KB
references/finetuning/references/rlaif_guide.mdmarkdown3.8 KB
references/finetuning/references/rlvr_reward_function.mdmarkdown10.0 KB
references/finetuning/scripts/mlflow_reference.pyscript841 B
references/finetuning/templates/nova_rlvr_reward_function_source_template.pyscript12.8 KB
references/finetuning/templates/rlvr_reward_function_source_template.pyscript9.1 KB
references/manage-mlflow/overview.mdmarkdown4.8 KB
references/manage-mlflow/references/app-creation-workflow.mdmarkdown3.2 KB
references/manage-mlflow/references/app-deletion-workflow.mdmarkdown2.3 KB
references/manage-mlflow/references/app-discovery-workflow.mdmarkdown1.5 KB
references/manage-mlflow/references/app-update-workflow.mdmarkdown1.7 KB
references/manage-mlflow/references/sm-mlflow-guide.mdmarkdown1.3 KB
references/model-deployment/code_templates/deploy-jumpstart-sagemaker.pyscript7.5 KB
references/model-deployment/code_templates/deploy-nova-bedrock.pyscript2.0 KB
references/model-deployment/code_templates/deploy-nova-sagemaker.pyscript1.9 KB
references/model-deployment/code_templates/deploy-oss-bedrock.pyscript4.0 KB
references/model-deployment/code_templates/deploy-oss-sagemaker.pyscript2.6 KB
references/model-deployment/code_templates/optimize-benchmark.pyscript4.6 KB
references/model-deployment/code_templates/optimize-huggingface-to-s3.pyscript2.0 KB
references/model-deployment/code_templates/optimize-recommendation.pyscript6.4 KB
references/model-deployment/overview.mdmarkdown16.6 KB
references/model-deployment/references/code_output_guide.mdmarkdown3.2 KB
references/model-deployment/references/deploy-jumpstart-sagemaker.mdmarkdown14.3 KB
references/model-deployment/references/deploy-nova-bedrock.mdmarkdown4.4 KB
references/model-deployment/references/deploy-nova-sagemaker.mdmarkdown5.8 KB
references/model-deployment/references/deploy-oss-bedrock.mdmarkdown5.2 KB
references/model-deployment/references/deploy-oss-sagemaker.mdmarkdown12.9 KB
references/model-deployment/references/model-licenses.mdmarkdown11.9 KB
references/model-deployment/references/optimization/benchmark-workflow.mdmarkdown12.7 KB
references/model-deployment/references/optimization/benchmarking-guidance.mdmarkdown8.9 KB
references/model-deployment/references/optimization/huggingface-to-s3.mdmarkdown3.1 KB
references/model-deployment/references/optimization/interpreting-results.mdmarkdown8.5 KB
references/model-deployment/references/optimization/recommendation-deploy.mdmarkdown1.5 KB
references/model-deployment/references/optimization/recommendation-options.mdmarkdown3.1 KB
references/model-deployment/references/optimization/recommendation-workflow.mdmarkdown10.3 KB
references/model-evaluation/code_templates/custom_scorer_evaluator.pyscript2.5 KB
references/model-evaluation/code_templates/llmaaj_evaluator.pyscript2.8 KB
references/model-evaluation/overview.mdmarkdown5.2 KB
references/model-evaluation/references/code_output_guide.mdmarkdown3.3 KB
references/model-evaluation/references/create-reward-function.mdmarkdown3.0 KB
references/model-evaluation/references/custom-lambda-scorer.mdmarkdown4.4 KB
references/model-evaluation/references/custom-scorer-evaluation.mdmarkdown10.5 KB
references/model-evaluation/references/evaluation-type-guide.mdmarkdown10.1 KB
references/model-evaluation/references/llmaaj-builtin-evaluation.mdmarkdown5.0 KB
references/model-evaluation/references/llmaaj-custom-evaluation.mdmarkdown2.5 KB
references/model-evaluation/references/llmaaj-evaluation.mdmarkdown14.6 KB
references/model-evaluation/references/supported-judge-models.mdmarkdown2.9 KB
references/model-evaluation/scripts/nova_reward_function_source_template.pyscript13.1 KB
references/model-evaluation/scripts/reward_function_source_template.pyscript9.2 KB
references/model-evaluation/scripts/validate_custom_metrics.pyscript3.5 KB
references/model-selection/overview.mdmarkdown10.3 KB
references/model-selection/references/benchmarks/agentic-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/coding-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/gpqa.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/hle.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/ifbench.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/intelligence-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/mmmu-pro.mdmarkdown3.0 KB
references/model-selection/references/benchmarks/tau2.mdmarkdown2.8 KB
references/model-selection/references/model-licenses.mdmarkdown7.3 KB
references/model-selection/references/model-selection.mdmarkdown8.2 KB
references/model-selection/references/select-for-deployment.mdmarkdown19.8 KB
references/model-selection/scripts/filter_deployable_models.pyscript10.1 KB
references/model-selection/scripts/get_deployable_models.pyscript21.7 KB
references/model-selection/scripts/get_model_names.pyscript1.4 KB
references/model-selection/scripts/search_deployable_models.pyscript34.9 KB
references/planning/overview.mdmarkdown10.3 KB
references/planning/references/deploy-base-model-plan.mdmarkdown699 B
references/planning/references/evaluate-first-plan.mdmarkdown1.8 KB
references/planning/references/input-output-contracts.mdmarkdown3.1 KB
references/planning/references/model-customization-plan.mdmarkdown1.5 KB
references/planning/references/region-availability.mdmarkdown1.2 KB
references/planning/references/skill-routing-constraints.mdmarkdown2.7 KB
references/processing-jobs/code_templates/dataset_definition_processing.pyscript4.1 KB
references/processing-jobs/code_templates/framework_processing.pyscript3.0 KB
references/processing-jobs/code_templates/script_processing.pyscript2.9 KB
references/processing-jobs/code_templates/spark_processing.pyscript3.7 KB
references/processing-jobs/overview.mdmarkdown6.7 KB
references/processing-jobs/references/configuration.mdmarkdown4.6 KB
references/processing-jobs/references/troubleshooting.mdmarkdown2.4 KB
references/sdk-getting-started/overview.mdmarkdown643 B
references/sdk-getting-started/references/execution-role-setup.mdmarkdown2.4 KB
references/sdk-getting-started/references/sagemaker-python-sdk-setup.mdmarkdown2.1 KB
references/training-jobs/code_templates/custom_training.pyscript6.2 KB
references/training-jobs/code_templates/jumpstart_training.pyscript5.6 KB
references/training-jobs/code_templates/recipe_training.pyscript5.7 KB
references/training-jobs/overview.mdmarkdown9.1 KB
references/training-jobs/references/configuration.mdmarkdown4.0 KB
references/training-jobs/references/troubleshooting.mdmarkdown4.4 KB
references/use-case-specification/overview.mdmarkdown2.0 KB
references/use-case-specification/references/spec-for-deployment.mdmarkdown3.9 KB
references/use-case-specification/references/spec-for-finetuning.mdmarkdown2.0 KB

references/model-deployment/references/optimization/interpreting-results.md

View on GitHub
# Interpreting Results

Guide for presenting benchmark and recommendation results to users.

## Benchmark Job Results

`job.show_result()` returns a `BenchmarkResult` (it downloads and parses the job's `output.tar.gz` from S3). Fill the fields below from it: `print(result)` renders the key metrics as a table, typed metrics are on `result.metrics` (each with p50/p90/p99 and average where applicable), and the raw profile is on `result.profile`.

Report results **consistently**. Always lead with this one-line headline, filled from the run, then follow with second-order metrics. Do not free-form the summary.

**Headline (fill this template exactly):**

> Serving **{concurrency}** concurrent requests, each request completes between **{request_latency_p50_seconds}s (p50)** and **{request_latency_{tail}_seconds}s ({tail})**.

Convert `request_latency` from ms to seconds (1 decimal). Example:

> Serving **4** concurrent requests, each request completes between **2.2s (p50)** and **2.7s (p99)**.

**Pick the tail percentile `{tail}` from the run's sample count (`request_count`)** so you never lead with an under-sampled number (see `benchmarking-guidance.md` → "How many requests do I need?"):

- `request_count` ≥ ~1000 → use **p99**.
- ~200–1000 → use **p90** (say "p90" in the headline, not p99).
- < ~200 → use **p50 only** (drop the tail clause) and note the run is too small for a trustworthy tail.

The default `request_count` is 100, so a default run leads with p50 and no tail unless the user asked for more requests. When you report a tail percentile, state the sample count alongside it so the reader can judge confidence.

**Then** the second-order metrics (these come *after* the headline, never instead of it):

| Metric | Stat | Value | Unit |
| ------ | ---- | ----- | ---- |
| OutputTokenThroughput | avg | {value} | tokens/s |
| RequestThroughput | avg | {value} | requests/s |
| TimeToFirstToken | p50 / p99 | {value} / {value} | ms |
| InterTokenLatency | p50 | {value} | ms |
| InputSequenceLength | avg | {value} | tokens |
| OutputSequenceLength | avg | {value} | tokens |

**Always include InputSequenceLength and OutputSequenceLength** (from `input_sequence_length` / `output_sequence_length` in the result). They characterize the actual traffic that produced these numbers — latency and throughput are only comparable across runs at the same sequence lengths. This is what makes a **synthetic vs. dataset comparison** meaningful: it lets the reader see *how much* the real traffic differed from the synthetic profile, so the latency delta is attributable.

> **Note:** Values are illustrative placeholders — fill them from the actual `job.show_result()` output. Results vary by model size, instance type, concurrency, and workload.

### Comparing two or more benchmark runs

When the user has more than one run to weigh against each other — synthetic vs. dataset, before vs. after an optimization, instance A vs. B, or a concurrency sweep — do **not** hand-diff the tables. Use the first-class `compare_benchmarks(...)` utility; it aligns every metric across runs and reports a signed percentage change oriented so **`+` is always better** (higher throughput / lower latency), so the reader never has to reason about sign direction per metric.

The code for this is **Cell 7 of `../../code_templates/optimize-benchmark.py`** — generate that cell rather than writing the call inline. It takes the current run's result plus a baseline reloaded with `BenchmarkJob.get("<job-name>").show_result()` (the FIRST argument is the baseline), and accepts `stat="avg"` (or `"p50"`/`"p90"`/`"p95"`/`"p99"`/`"min"`/`"max"`).

Present the printed table as-is; then call out the one or two metrics that moved most (e.g. "+305% throughput but −719% latency at concurrency 150 — the endpoint is saturated"). `compare_benchmarks` rejects concurrency-search/sweep results (they have no single metric profile) — compare the individual winning runs instead. Because deltas are only meaningful at matched sequence lengths, keep ISL/OSL in view (see the note above).

Key insights to highlight (optional, after the numbers):

- **TTFT p50 vs p99** — a large gap means inconsistent latency (batching / cold starts).
- **OutputTokenThroughput** — higher is better for batch workloads.
- **Concurrency impact** — if multiple runs at different concurrency were done, compare how latency and throughput scale.
- **Synthetic vs. dataset** — when comparing a synthetic run to a dataset run on the same endpoint, put the two runs' input/output sequence lengths side by side. If the dataset's sequence lengths differ from the synthetic profile, that difference explains the latency delta — quantify it (e.g. "dataset averaged 180/95 in/out tokens vs the synthetic 128/128, so the higher latency is expected").
- **Cost framing** — benchmark results measure *performance*, not cost. To frame cost, pair OutputTokenThroughput with the instance's on-demand hourly rate (retrieve via `aws pricing get-products --service-code AmazonSageMaker --filters ...` — never fabricate) to express $/1M tokens. If the endpoint uses inference components (IC), the backing instance type may not be directly visible — tell the user, and retrieve pricing once you identify the instance (e.g. from `describe-endpoint-config`). If pricing cannot be retrieved at all, say so upfront and omit the column rather than failing silently.

## Recommendation Job Results

Present recommendations as a ranked table:

> "Here are the recommendations, ranked by [cost/latency/throughput]:
>
> | # | Instance Type  | TTFT p50 | Throughput | Optimizations | Est. Cost |
> | - | -------------- | -------- | ---------- | ------------- | --------- |
> | 1 | ml.g6.xlarge   | 95ms     | 42 tok/s   | Kernel Tuning | <retrieve via aws pricing get-products> |
> | 2 | ml.g5.xlarge   | 110ms    | 38 tok/s   | None          | <retrieve via aws pricing get-products> |
> | 3 | ml.g6.12xlarge | 45ms     | 120 tok/s  | Kernel Tuning | <retrieve via aws pricing get-products> |

> **Note:** The table above is an illustrative example, not an exhaustive list of possible results. Actual recommendations depend on the model, performance target, and available instance types.
>
> ⚠️ **Always** retrieve real-time pricing with `aws pricing get-products --service-code AmazonSageMaker --filters ...` before populating the Est. Cost column. Never estimate or fabricate cost values. If pricing cannot be retrieved, omit the column or show "N/A".

### What Each Field Means

- **Instance Type** — The GPU instance the model was tested on
- **TTFT (Time to First Token)** — How long until the first token is generated. Lower is better for interactive use cases.
- **Throughput (Output Token Throughput)** — Tokens generated per second. Higher is better for batch processing.
- **Optimizations** — What the service applied:
  - **Kernel Tuning** — GPU kernel optimizations specific to the hardware. Improves throughput with no quality impact.
  - **Speculative Decoding** — Uses a smaller draft model to predict tokens ahead. Improves latency but requires a compatible draft model.
- **Est. Cost** — The on-demand hourly price for the instance type. The recommendation job does not calculate cost directly — retrieve current rates with `aws pricing get-products --service-code AmazonSageMaker --filters ...` (see guardrail above). Actual costs depend on usage patterns, reserved capacity, and Savings Plans.
- **CopyCountPerInstance** — How many model copies fit on one instance. More copies = higher throughput per instance.

### Helping the User Choose

Guide based on their performance target:

- **Cost target** → Recommend #1 (lowest cost that meets baseline performance)
- **Latency target** → Recommend the one with lowest TTFT at the user's target percentile
- **Throughput target** → Recommend the one with highest OutputTokenThroughput

If the user is unsure, suggest:

> "For interactive applications (chatbots, real-time), prioritize low TTFT.
> For batch processing (summarization, translation), prioritize high throughput.
> For cost-sensitive workloads, the cost-optimized recommendation balances both."

### ModelPackage Deployment

Every recommendation is backed by a deployable ModelPackage (exposed on the row's `model_details`). Explain:

> "Each recommendation is packaged as a deployable ModelPackage. You can deploy any of them directly with `model_builder.deploy(recommendation_index=N)` — the model weights, container image, environment variables, and optimization artifacts are all bundled together.
>
> Would you like me to generate deployment code for one of these recommendations?"
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall