Skip to content

opensite-ai/opensite-skills / aws-ai-ml

aws-ai-ml

Verified

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md8.1 KB
references/dataset-evaluation/overview.mdmarkdown5.4 KB
references/dataset-evaluation/references/custom-scorer-evaluation-dataset-formats.mdmarkdown3.6 KB
references/dataset-evaluation/references/strategy_data_requirements.mdmarkdown4.8 KB
references/dataset-evaluation/scripts/format_detector.pyscript26.7 KB
references/dataset-transformation/code_templates/transformation.pyscript1.5 KB
references/dataset-transformation/overview.mdmarkdown13.3 KB
references/dataset-transformation/references/code_output_guide.mdmarkdown2.8 KB
references/dataset-transformation/references/dataset_transformation_code.mdmarkdown5.2 KB
references/dataset-transformation/references/sagemaker_dataset_formats.mdmarkdown2.9 KB
references/dataset-transformation/scripts/transformation_tools.pyscript4.8 KB
references/directory-management/overview.mdmarkdown1.9 KB
references/endpoint-diagnostics/overview.mdmarkdown3.5 KB
references/endpoint-diagnostics/references/minimum_iam_policy.mdmarkdown1.2 KB
references/endpoint-diagnostics/scripts/collect_diagnostics.pyscript12.8 KB
references/finetuning-technique/overview.mdmarkdown2.1 KB
references/finetuning-technique/references/finetune_technique_selection_guide.mdmarkdown2.3 KB
references/finetuning-technique/scripts/get_recipes.pyscript817 B
references/finetuning/code_templates/dpo.pyscript6.4 KB
references/finetuning/code_templates/rlaif_builtin.pyscript5.8 KB
references/finetuning/code_templates/rlaif_custom_prompt.pyscript6.3 KB
references/finetuning/code_templates/rlvr.pyscript6.5 KB
references/finetuning/code_templates/sft.pyscript6.1 KB
references/finetuning/overview.mdmarkdown11.4 KB
references/finetuning/references/code_output_guide.mdmarkdown3.2 KB
references/finetuning/references/continuous_customization.mdmarkdown8.4 KB
references/finetuning/references/eula_links.mdmarkdown7.3 KB
references/finetuning/references/rlaif_guide.mdmarkdown3.8 KB
references/finetuning/references/rlvr_reward_function.mdmarkdown10.0 KB
references/finetuning/scripts/mlflow_reference.pyscript841 B
references/finetuning/templates/nova_rlvr_reward_function_source_template.pyscript12.8 KB
references/finetuning/templates/rlvr_reward_function_source_template.pyscript9.1 KB
references/manage-mlflow/overview.mdmarkdown4.8 KB
references/manage-mlflow/references/app-creation-workflow.mdmarkdown3.2 KB
references/manage-mlflow/references/app-deletion-workflow.mdmarkdown2.3 KB
references/manage-mlflow/references/app-discovery-workflow.mdmarkdown1.5 KB
references/manage-mlflow/references/app-update-workflow.mdmarkdown1.7 KB
references/manage-mlflow/references/sm-mlflow-guide.mdmarkdown1.3 KB
references/model-deployment/code_templates/deploy-jumpstart-sagemaker.pyscript7.5 KB
references/model-deployment/code_templates/deploy-nova-bedrock.pyscript2.0 KB
references/model-deployment/code_templates/deploy-nova-sagemaker.pyscript1.9 KB
references/model-deployment/code_templates/deploy-oss-bedrock.pyscript4.0 KB
references/model-deployment/code_templates/deploy-oss-sagemaker.pyscript2.6 KB
references/model-deployment/code_templates/optimize-benchmark.pyscript4.6 KB
references/model-deployment/code_templates/optimize-huggingface-to-s3.pyscript2.0 KB
references/model-deployment/code_templates/optimize-recommendation.pyscript6.4 KB
references/model-deployment/overview.mdmarkdown16.6 KB
references/model-deployment/references/code_output_guide.mdmarkdown3.2 KB
references/model-deployment/references/deploy-jumpstart-sagemaker.mdmarkdown14.3 KB
references/model-deployment/references/deploy-nova-bedrock.mdmarkdown4.4 KB
references/model-deployment/references/deploy-nova-sagemaker.mdmarkdown5.8 KB
references/model-deployment/references/deploy-oss-bedrock.mdmarkdown5.2 KB
references/model-deployment/references/deploy-oss-sagemaker.mdmarkdown12.9 KB
references/model-deployment/references/model-licenses.mdmarkdown11.9 KB
references/model-deployment/references/optimization/benchmark-workflow.mdmarkdown12.7 KB
references/model-deployment/references/optimization/benchmarking-guidance.mdmarkdown8.9 KB
references/model-deployment/references/optimization/huggingface-to-s3.mdmarkdown3.1 KB
references/model-deployment/references/optimization/interpreting-results.mdmarkdown8.5 KB
references/model-deployment/references/optimization/recommendation-deploy.mdmarkdown1.5 KB
references/model-deployment/references/optimization/recommendation-options.mdmarkdown3.1 KB
references/model-deployment/references/optimization/recommendation-workflow.mdmarkdown10.3 KB
references/model-evaluation/code_templates/custom_scorer_evaluator.pyscript2.5 KB
references/model-evaluation/code_templates/llmaaj_evaluator.pyscript2.8 KB
references/model-evaluation/overview.mdmarkdown5.2 KB
references/model-evaluation/references/code_output_guide.mdmarkdown3.3 KB
references/model-evaluation/references/create-reward-function.mdmarkdown3.0 KB
references/model-evaluation/references/custom-lambda-scorer.mdmarkdown4.4 KB
references/model-evaluation/references/custom-scorer-evaluation.mdmarkdown10.5 KB
references/model-evaluation/references/evaluation-type-guide.mdmarkdown10.1 KB
references/model-evaluation/references/llmaaj-builtin-evaluation.mdmarkdown5.0 KB
references/model-evaluation/references/llmaaj-custom-evaluation.mdmarkdown2.5 KB
references/model-evaluation/references/llmaaj-evaluation.mdmarkdown14.6 KB
references/model-evaluation/references/supported-judge-models.mdmarkdown2.9 KB
references/model-evaluation/scripts/nova_reward_function_source_template.pyscript13.1 KB
references/model-evaluation/scripts/reward_function_source_template.pyscript9.2 KB
references/model-evaluation/scripts/validate_custom_metrics.pyscript3.5 KB
references/model-selection/overview.mdmarkdown10.3 KB
references/model-selection/references/benchmarks/agentic-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/coding-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/gpqa.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/hle.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/ifbench.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/intelligence-index.mdmarkdown2.8 KB
references/model-selection/references/benchmarks/mmmu-pro.mdmarkdown3.0 KB
references/model-selection/references/benchmarks/tau2.mdmarkdown2.8 KB
references/model-selection/references/model-licenses.mdmarkdown7.3 KB
references/model-selection/references/model-selection.mdmarkdown8.2 KB
references/model-selection/references/select-for-deployment.mdmarkdown19.8 KB
references/model-selection/scripts/filter_deployable_models.pyscript10.1 KB
references/model-selection/scripts/get_deployable_models.pyscript21.7 KB
references/model-selection/scripts/get_model_names.pyscript1.4 KB
references/model-selection/scripts/search_deployable_models.pyscript34.9 KB
references/planning/overview.mdmarkdown10.3 KB
references/planning/references/deploy-base-model-plan.mdmarkdown699 B
references/planning/references/evaluate-first-plan.mdmarkdown1.8 KB
references/planning/references/input-output-contracts.mdmarkdown3.1 KB
references/planning/references/model-customization-plan.mdmarkdown1.5 KB
references/planning/references/region-availability.mdmarkdown1.2 KB
references/planning/references/skill-routing-constraints.mdmarkdown2.7 KB
references/processing-jobs/code_templates/dataset_definition_processing.pyscript4.1 KB
references/processing-jobs/code_templates/framework_processing.pyscript3.0 KB
references/processing-jobs/code_templates/script_processing.pyscript2.9 KB
references/processing-jobs/code_templates/spark_processing.pyscript3.7 KB
references/processing-jobs/overview.mdmarkdown6.7 KB
references/processing-jobs/references/configuration.mdmarkdown4.6 KB
references/processing-jobs/references/troubleshooting.mdmarkdown2.4 KB
references/sdk-getting-started/overview.mdmarkdown643 B
references/sdk-getting-started/references/execution-role-setup.mdmarkdown2.4 KB
references/sdk-getting-started/references/sagemaker-python-sdk-setup.mdmarkdown2.1 KB
references/training-jobs/code_templates/custom_training.pyscript6.2 KB
references/training-jobs/code_templates/jumpstart_training.pyscript5.6 KB
references/training-jobs/code_templates/recipe_training.pyscript5.7 KB
references/training-jobs/overview.mdmarkdown9.1 KB
references/training-jobs/references/configuration.mdmarkdown4.0 KB
references/training-jobs/references/troubleshooting.mdmarkdown4.4 KB
references/use-case-specification/overview.mdmarkdown2.0 KB
references/use-case-specification/references/spec-for-deployment.mdmarkdown3.9 KB
references/use-case-specification/references/spec-for-finetuning.mdmarkdown2.0 KB

references/model-deployment/code_templates/optimize-benchmark.py

View on GitHub
# Cell 0 [markdown]: Inference Optimization — Benchmark an Endpoint

# Cell 1: Setup

# sagemaker>=3.20.0 ships the GenAI benchmarking interface (start_benchmark / Workload / job.show_result).
# %pip install --upgrade "sagemaker>=3.20.0,<4.0" --quiet  # NOTEBOOK_ONLY

# Cell 2: Configuration

import json
import os

os.environ["AWS_DEFAULT_REGION"] = "[REGION]"

from sagemaker.core import Attribution, set_attribution
from sagemaker.serve import start_benchmark
from sagemaker.serve.ai_inference_recommender import Workload

set_attribution(Attribution.SAGEMAKER_AGENT_PLUGIN)

ENDPOINT_NAME = "[ENDPOINT_NAME]"
ROLE_ARN = "[ROLE_ARN]"
S3_OUTPUT_LOCATION = "[S3_OUTPUT_LOCATION]"
BENCHMARK_JOB_NAME = "[BENCHMARK_JOB_NAME]"
WORKLOAD_CONFIG_NAME = "[WORKLOAD_CONFIG_NAME]"
TOKENIZER = "[TOKENIZER]"

# Cell 3: Define the Workload

# Default: synthetic prompts against an OpenAI ChatCompletions endpoint.
# For a load test, raise concurrency (e.g. 50-200) and request_count.
workload = Workload.synthetic(
    tokenizer=TOKENIZER,
    prompt_input_tokens_mean=[INPUT_TOKENS_MEAN],
    prompt_input_tokens_stddev=[INPUT_TOKENS_STDDEV],
    output_tokens_mean=[OUTPUT_TOKENS_MEAN],
    output_tokens_stddev=[OUTPUT_TOKENS_STDDEV],
    concurrency=[CONCURRENCY],
    request_count=[REQUEST_COUNT],
    # Optional extra AIPerf params passthrough — e.g. ignore_eos when an exact
    # output length is required, a low sampling temperature for speculative-decoding
    # runs, or a warmup control if the runtime exposes one. Omit if not needed.
    # **[EXTRA_PARAMS]  # e.g. {"ignore_eos": True, "temperature": 0.2}
)

# Custom (non-OpenAI) endpoint format — provide a Jinja2 request template and a
# JMESPath response_field (omit response_field to let AIPerf auto-detect):
#   workload = Workload.template(
#       "[REQUEST_TEMPLATE]",  # local path or inline Jinja2 string
#       response_field="[RESPONSE_FIELD]",  # e.g. "generated_text"
#       tokenizer=TOKENIZER, concurrency=[CONCURRENCY], request_count=[REQUEST_COUNT])
#
# Benchmark on your own data / real traffic — replay a dataset or SageMaker Data
# Capture. custom_dataset_type is an OPTIONAL hint naming the schema so AIPerf
# parses it ("openai-chat", "openai-completions", "sharegpt", or
# "sagemaker-datacapture"); omit it to let AIPerf auto-detect. Inspect the data
# to pick the value; if it's an unsupported format, convert it first with the
# dataset-transformation skill.
#   workload = Workload.from_dataset(
#       "[DATASET_S3_URI]", tokenizer=TOKENIZER,
#       custom_dataset_type="openai-chat",  # optional; from the schema you detected
#       concurrency=[CONCURRENCY], request_count=[REQUEST_COUNT])

# Cell 4: Run the Benchmark

job = start_benchmark(
    endpoint=ENDPOINT_NAME,
    workload=workload,
    role=ROLE_ARN,
    output_path=S3_OUTPUT_LOCATION,
    name=BENCHMARK_JOB_NAME,
    workload_config_name=WORKLOAD_CONFIG_NAME,
    # inference_components=["[INFERENCE_COMPONENT_NAME]"],  # only if the endpoint uses inference components
    wait=True,
)
print(f"Benchmark job {job.get_name()} finished: {job.ai_benchmark_job_status}")

# Cell 5: Display Results

result = job.show_result()
print(result)

# Cell 6: Save Manifest
# Save manifest - record output of workflow step for future reference
from pathlib import Path
manifest_dir = Path("[PROJECT_DIR]") / "manifests"
manifest_dir.mkdir(parents=True, exist_ok=True)
manifest_path = manifest_dir / f"benchmark-{BENCHMARK_JOB_NAME}.json"
manifest_path.write_text(json.dumps({
    "benchmark_job_name": BENCHMARK_JOB_NAME,
    "endpoint_name": ENDPOINT_NAME,
    "workload_config_name": WORKLOAD_CONFIG_NAME,
    "s3_output_location": result.s3_output_location,
    "status": job.ai_benchmark_job_status,
}, indent=2))
print(f"Manifest saved: {manifest_path}")

# Cell 7: Compare Runs (optional)
# Include this cell only when comparing this run against another — before/after an
# optimization, synthetic vs. dataset, or a step in a concurrency sweep. compare_benchmarks
# aligns every metric across runs and reports a signed Δ% oriented so "+" is always better
# (higher throughput / lower latency). The FIRST result is the baseline. Reload a prior run
# with BenchmarkJob.get("<job-name>").show_result().
from sagemaker.serve.ai_inference_recommender import compare_benchmarks
from sagemaker.serve.ai_inference_recommender.jobs import BenchmarkJob

baseline_result = BenchmarkJob.get("[BASELINE_JOB_NAME]").show_result()
comparison = compare_benchmarks(
    baseline_result,
    result,
    names=["[BASELINE_LABEL]", "[THIS_RUN_LABEL]"],
    stat="avg",  # or "p50" / "p90" / "p95" / "p99" / "min" / "max"
)
print(comparison)
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall