hyperlane-xyz/hyperlane-monorepo / .agents/skills/fix-merkle-root-mismatch
fix-merkle-root-mismatch
VerifiedApply fixes for merkle root mismatch issues in the relayer database. Use after running /investigate-merkle-root-mismatch to investigate and identify the mismatched entries. Requires user confirmation before any database modifications.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.2 KB |
SKILL.md
View on GitHub---
name: fix-merkle-root-mismatch
description:
Apply fixes for merkle root mismatch issues in the relayer database.
Use after running /investigate-merkle-root-mismatch to investigate and identify
the mismatched entries. Requires user confirmation before any database modifications.
---
# Fix Merkle Root Mismatch
## When to Use
Use this skill AFTER running `/investigate-merkle-root-mismatch` to investigate the issue.
1. **User request triggers:**
- "Fix the merkle root mismatch on [chain]"
- "Apply the merkle tree fixes"
- "Update the relayer database with correct message IDs"
## Input Parameters
| Parameter | Required | Default | Description |
| ------------------------ | -------- | ---------- | --------------------------------------------------------------- |
| `origin` | Yes | - | The origin chain to fix (e.g., `polygonzkevm`, `ethereum`) |
| `domain_id` | Yes | - | Domain ID of the chain |
| `environment` | No | `mainnet3` | `mainnet3` or `testnet4` |
| `merkle_tree_insertions` | Yes | - | List of corrections: `[{leaf_index, message_id, block_number}]` |
## Prerequisites
1. **Investigation completed** - Run `/investigate-merkle-root-mismatch` first to identify mismatched entries
2. `kubectl` access to the relayer pods
3. Port-forward already established to relayer (port 9090)
## Fix Workflow
### Step 0: Find Monorepo Root
```bash
MONOREPO_ROOT=$(git rev-parse --show-toplevel)
```
### Step 1: Verify Port-Forward is Active
Check if port 9090 is already in use:
```bash
lsof -i :9090
```
If not in use, start port-forward in background:
```bash
kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090:9090 -n [environment] &
```
Wait a few seconds, then verify it's working:
```bash
curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=0&leaf_index_end=1"
```
### Step 2: Present Changes for Confirmation
**IMPORTANT: User confirmation required before executing.**
Before making any changes to the relayer database, you MUST:
1. Present a summary of ALL changes to be made in a clear table format:
| Leaf Index | Correct Message ID | Block Number |
| ---------- | ------------------------------------------------------------------ | ------------ |
| 1477 | 0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7 | 9891832 |
| 1478 | 0xa66dbdc1874acfddf14e75e6a20dd1632e9e3206c5f5855884da0d26d8ca12fa | 9891872 |
2. Ask the user to get explicit confirmation:
- Question: "Do you want to apply these merkle tree fixes to the relayer database?"
- Options: "Yes, apply fixes" / "No, cancel"
3. **Only proceed if the user explicitly confirms.**
### Step 3: Apply Merkle Tree Fixes
Once confirmed, insert the correct merkle tree insertions via the relayer API:
```bash
curl -X POST \
-H 'Content-type: application/json' \
'localhost:9090/merkle_tree_insertions' \
-d '{
"merkle_tree_insertions": [
{
"chain": [domain_id],
"insertion_block_number": [block_number],
"leaf_index": [index],
"message_id": "0x..."
}
]
}'
```
**Request schema:**
```json
{
"merkle_tree_insertions": [
{
"chain": "<domain_id as number>",
"insertion_block_number": "<block number as number>",
"leaf_index": "<leaf index as number>",
"message_id": "<0x-prefixed H256 hash>"
}
]
}
```
**Verify response:** The API should return a success response with the count of inserted entries.
### Step 4: Restart Relayer
**IMPORTANT: User confirmation required before executing.**
Ask the user to confirm:
- Question: "Do you want to restart the relayer to apply the database fixes?"
- Options: "Yes, restart relayer" / "No, I'll restart manually later"
**Only proceed if the user explicitly confirms.**
Once confirmed, restart the relayer to rebuild its in-memory merkle tree:
```bash
pnpm --dir typescript/infra exec tsx ./scripts/agents/restart-agents.ts -e [environment] --context hyperlane --role relayer
```
### Step 5: Validate Fix
After the relayer restarts, verify the fix:
#### 5.1: Get Latest Index
Query Grafana for the current tree size:
```
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_latest_tree_insertion_index{origin="[origin]", hyperlane_deployment="[environment]"}
- startTime: now-1h
- queryType: instant
```
#### 5.2: Spot-Check Roots at Key Indices
Compare validator and relayer roots at three points: first fixed index, middle, and latest.
```bash
index=[INDEX]
validator_root=$(curl -s "https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_${index}_with_id.json" | jq -r '.value.checkpoint.root')
relayer_root=$(curl -s "localhost:9090/merkle_proofs?domain_id=[domain_id]&leaf_index=${index}&root_index=${index}" | jq -r '.root')
echo "Index $index:"
echo " Validator: $validator_root"
echo " Relayer: 0x$relayer_root"
if [ "$validator_root" = "0x$relayer_root" ]; then echo " ✓ Match"; else echo " ❌ MISMATCH"; fi
```
Run this for:
1. First fixed index (e.g., 37352)
2. Middle index (e.g., 39000)
3. Latest index (e.g., 41172)
#### 5.3: Verify Mismatch Metric is Clear
```
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_merkle_root_mismatch{origin="[origin]"}
- startTime: now-1h
- queryType: instant
```
**Expected result:** Empty result or value of 0.
#### 5.4: Report Results
Present validation results in a table:
| Index | Location | Result |
| ----- | ----------- | ------- |
| 37352 | First fixed | ✓ Match |
| 39000 | Middle | ✓ Match |
| 41172 | Latest | ✓ Match |
If still mismatched:
- Inform the user that additional entries may need fixing
- Suggest re-running `/investigate-merkle-root-mismatch` to investigate further
## API Reference
### Relayer Endpoints
| Endpoint | Method | Description |
| ------------------------- | ------ | ----------------------------------- |
| `/merkle_tree_insertions` | GET | List merkle tree insertions from DB |
| `/merkle_tree_insertions` | POST | Insert/update merkle tree entries |
### Request/Response Examples
**POST /merkle_tree_insertions**
Request:
```json
{
"merkle_tree_insertions": [
{
"chain": 1101,
"insertion_block_number": 9891832,
"leaf_index": 1477,
"message_id": "0x4bb3e20db45366a6a360ad2639c5421ea622a69b89b2edb045aa97e2051529b7"
}
]
}
```
Response (success):
```json
{
"status": "success",
"data": {
"count": 1
}
}
```
## Common Issues
1. **Port-forward disconnected:** Re-establish before applying fixes
2. **API returns error:** Check the error message, verify domain_id and message_id format
3. **Relayer restart fails:** Check kubectl access and pod status
4. **Fix didn't work:** May have missed some mismatched indices, re-investigate
## Runbook Reference
Full runbook: https://www.notion.so/hyperlanexyz/Merkle-Root-Mismatch-26a6d35200d680a2857dcd0b228d4ab7
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |