Skip to content

opensite-ai/opensite-skills / amazon-bedrock

amazon-bedrock

Flagged

Builds generative AI applications on Amazon Bedrock. Covers model invocation (Converse API, InvokeModel), RAG with Knowledge Bases, Bedrock Agents, Guardrails, and AgentCore (including the Harness managed agent loop). Applies when invoking models, setting up Knowledge Bases, creating agents, applying guardrails, deploying to AgentCore, migrating/porting/converting a Bedrock Agent (including inline agents) to an AgentCore Harness, troubleshooting Bedrock errors (ThrottlingException, AccessDeniedException), or choosing Bedrock models (Claude, Llama, Nova, Titan). Also for prompt caching, quota and throttling diagnosis, cost tracking, migrating between Claude model generations, chunking strategies, and Bedrock model selection. Covers AgentCore Payments (x402, microtransactions, Payment Manager, Connector, Instrument, Coinbase CDP, Stripe Privy, paid endpoints, agent payments). NOT for custom model training, Rekognition, or Comprehend. For SageMaker model discovery, customization, or endpoints, use aws-ai-ml.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md34.1 KB
assets/kb_shim.py.tmplother3.4 KB
assets/lambda_shim.py.tmplother8.0 KB
assets/tool_schema.json.tmplother389 B
references/agentcore-credentials-and-security.mdmarkdown6.5 KB
references/agentcore-gateway.mdmarkdown7.0 KB
references/agentcore-harness.mdmarkdown22.0 KB
references/agentcore-memory-observability.mdmarkdown5.6 KB
references/agentcore-payments-setup-script.mdmarkdown12.3 KB
references/agentcore-payments-wiring.mdmarkdown12.1 KB
references/agentcore-payments.mdmarkdown19.5 KB
references/agentcore-registry-evaluations.mdmarkdown6.8 KB
references/agentcore-runtime-container-build.mdmarkdown10.4 KB
references/agentcore-runtime.mdmarkdown7.6 KB
references/agents-and-action-groups.mdmarkdown16.5 KB
references/bedrock-agents-to-agentcore-harness/cli.mdmarkdown2.0 KB
references/bedrock-agents-to-agentcore-harness/deploy.mdmarkdown12.0 KB
references/bedrock-agents-to-agentcore-harness/discovery.mdmarkdown3.9 KB
references/bedrock-agents-to-agentcore-harness/eligibility.mdmarkdown3.3 KB
references/bedrock-agents-to-agentcore-harness/mapping.mdmarkdown8.9 KB
references/cost-tracking.mdmarkdown4.5 KB
references/guardrails.mdmarkdown14.7 KB
references/knowledge-bases-decision-guide.mdmarkdown2.8 KB
references/knowledge-bases-retrieval.mdmarkdown8.3 KB
references/knowledge-bases-setup.mdmarkdown17.9 KB
references/managed-knowledge-bases-retrieval.mdmarkdown8.1 KB
references/managed-knowledge-bases-setup.mdmarkdown12.7 KB
references/migrate-bedrock-agents-to-agentcore-harness.mdmarkdown11.5 KB
references/model-invocation.mdmarkdown11.0 KB
references/model-migration.mdmarkdown4.3 KB
references/model-selection-guide.mdmarkdown7.4 KB
references/prompt-caching.mdmarkdown5.5 KB
references/prompt-engineering-by-model.mdmarkdown7.9 KB
references/quota-health.mdmarkdown4.3 KB
references/sdk-converse-api-python.mdmarkdown6.3 KB
references/sdk-converse-api-typescript.mdmarkdown6.5 KB
scripts/fetch_bedrock_agent.pyscript14.1 KB

references/agentcore-registry-evaluations.md

View on GitHub
# AgentCore Registry & Evaluations

## Table of Contents
- Agent Registry (Preview)
- Evaluations Service

> **Upcoming namespace migration**: The registry namespace is migrating from `bedrock-agentcore` to `agent-registry`. This will require updates to your endpoints, IAM policies, SDK, and CLI usage. Check the latest AWS documentation for the migration date and status.

## Agent Registry (Preview)

Catalog, discover, and govern AI agents and tools across an organization.

### Governance Workflow

The key non-obvious behavior — two modes:

| Mode | Behavior | Use For |
|------|----------|---------|
| **Auto-approve** | Records become discoverable immediately | Development environments (isolated accounts only) |
| **Manual approval** | Records require explicit approval before discovery | Production environments |

Status transitions: `DRAFT` → `PENDING_APPROVAL` → `APPROVED` (`REJECTED` reachable from `PENDING_APPROVAL`; `DEPRECATED` is the terminal state)

**Common failure**: Record stuck in `PENDING_APPROVAL` — governance workflow is set to manual approval but no one has approved. Check governance configuration or switch to auto-approve for dev.

### Registering Resources

Resource types: MCP servers, Agents (A2A agent card), agent skills, custom types.

**Constraints:**
- You MUST specify resource type, name, description, and invocation endpoint
- You MUST register: `aws bedrock-agentcore-control create-registry-record --registry-id <registry-id> --name <name> --descriptor-type <MCP|A2A|CUSTOM|AGENT_SKILLS> --description "<desc>"`
- Tags and capabilities metadata improve discoverability

### Searching and Discovery

- CLI: `aws bedrock-agentcore-control list-registry-records --registry-id <registry-id>`
- MCP endpoint: programmatic discovery via MCP protocol
- Filter by resource type, tags, capabilities

### Available Regions

Verify availability: `aws bedrock-agentcore-control list-registry-records --registry-id <registry-id> --region <region>`. Registry is a Preview feature — region availability is expanding.

## Evaluations Service

Automated agent quality assessment using LLM-as-a-Judge.

### Setup Workflow

```
Evaluation Setup:
- [ ] Step 1: Instrument agent with OTEL (see [memory & observability](agentcore-memory-observability.md))
- [ ] Step 2: Create evaluators (built-in or custom)
- [ ] Step 3: Configure online evaluation (sampling rate, data source)
- [ ] Step 4: Monitor scores in CloudWatch
```

### Built-in Evaluators

| Evaluator | What It Measures |
|-----------|-----------------|
| `Builtin.GoalSuccessRate` | Does the agent achieve the user's goal? |
| `Builtin.Coherence` | Is the response logically consistent? |
| `Builtin.Conciseness` | Is the response appropriately brief? |
| `Builtin.ContextRelevance` | Is the retrieved context relevant to the query? |
| `Builtin.Correctness` | Is the response factually correct? |
| `Builtin.Faithfulness` | Is the response grounded in provided context? |
| `Builtin.Harmfulness` | Does the response contain harmful content? |
| `Builtin.Helpfulness` | Does the response help the user? |
| `Builtin.InstructionFollowing` | Does the response follow the given instructions? |
| `Builtin.Refusal` | Does the response inappropriately refuse a valid request? |
| `Builtin.ResponseRelevance` | Is the response relevant to the query? |
| `Builtin.Stereotyping` | Does the response contain stereotyping? |
| `Builtin.ToolParameterAccuracy` | Are tool call parameters accurate? |
| `Builtin.ToolSelectionAccuracy` | Is the correct tool selected? |

Refer to the latest AWS documentation on AgentCore Evaluations built-in evaluators for the full current list.

### Custom Evaluators

Define your own evaluation criteria:
- Rubric: what constitutes a good/bad response for your use case
- Scoring scale: numeric (1-5) or binary (pass/fail)
- Custom prompt template: the LLM-as-a-Judge prompt

Create custom evaluators: `aws bedrock-agentcore-control create-evaluator --evaluator-name <name> --level <TOOL_CALL|TRACE|SESSION> --evaluator-config '{"llmAsAJudge":{"instructions":"<criteria>","ratingScale":{"numerical":[{"value":1,"label":"Poor","definition":"..."},{"value":5,"label":"Excellent","definition":"..."}]}}}'`

### Online vs On-Demand Evaluation

| Type | When | Use For |
|------|------|---------|
| **Online** | Continuous, samples production traffic | Monitoring quality over time |
| **On-demand** | Batch, against a test dataset | Regression testing, A/B comparison |

**Online evaluation constraints:**
- Configure sampling rate — evaluating every invocation is expensive (each evaluation is a model invocation)
- Start with 5-10% sampling, increase if quality issues detected
- Data source: which OTEL traces to evaluate

### Monitoring Scores

- Evaluation scores publish to CloudWatch automatically
- Create alarms for quality degradation: score drops below threshold
- Investigate low-scoring sessions: trace → evaluation result → root cause
- Create quality alarms — first discover the exact namespace (CloudWatch namespaces are case-sensitive):
  1. `aws cloudwatch list-metrics --namespace "Bedrock-AgentCore"` — if no results, try `--namespace "Bedrock-Agentcore"`
  2. Use the namespace that returns metrics in subsequent commands:

  `aws cloudwatch put-metric-alarm --alarm-name <name> --metric-name <metric> --namespace "<discovered-namespace>" --statistic Average --period 300 --threshold <value> --comparison-operator LessThanThreshold --evaluation-periods 3 --alarm-actions "<sns-topic-arn>"`

## Security Considerations

**Registry access control:**
- You MUST use least-privilege IAM policies — separate read (`list-registry-records`) from write (`create-registry-record`) permissions. Avoid `bedrock-agentcore:*`
- You MUST use IAM roles (not IAM users) for programmatic registry access
- You SHOULD add `aws:SourceArn` and `aws:SourceAccount` conditions to resource policies on registry resources
- You MUST restrict auto-approve governance mode to isolated development accounts — use manual approval in shared or production environments

**Evaluation data protection:**
- OTEL traces sent to evaluations contain user queries, agent responses, and tool call parameters — these may include PII
- You MUST ensure OTEL trace data is encrypted in transit (TLS) and at rest
- You SHOULD implement PII scrubbing in OTEL instrumentation before traces reach the evaluation service
- You MUST restrict access to evaluation results to authorized personnel only
- Encrypt CloudWatch log groups storing evaluation results with KMS

**Monitoring security:**
- You MUST encrypt SNS topics used for alarm actions with KMS
- You MUST validate that SNS topic subscribers are authorized to receive evaluation data
- You MUST enable CloudTrail for all `bedrock-agentcore-control` API calls — tracks who registered resources, who approved/rejected records, and who modified evaluations

- Refer to the latest AWS documentation on Bedrock AgentCore security best practices.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall