Skip to content

forcedotcom/sf-skills / plugins/builder/agentforce-adlc/skills/agentforce-observe

agentforce-observe

Verified

Analyze production Agentforce agent behavior using session traces and Data Cloud, and manage Agent Health Monitoring (AHM) alerts. TRIGGER when: user queries STDM session data or Data Cloud trace records; investigates production agent failures, regressions, or performance issues; asks about session traces, conversation logs, or agent metrics; wants to reproduce a reported production issue in preview; runs findSessions or trace analysis queries; creates, lists, updates, or deletes an AHM data alert on an agent metric (escalation rate, deflection, etc.); asks why an alert is not firing or wants to check whether alerts have fired (notification counts, or the per-alert Incidents view). DO NOT TRIGGER when: user creates, modifies, or debugs .agent files during development (use agentforce-generate); writes or runs test specs (use agentforce-test); uses sf agent preview for local development iteration; deploys or publishes agents.

references/ahm-alerts.md

View on GitHub
# AHM Alerts Reference — Agent Health Monitoring Data Alerts

End-to-end procedures for creating, listing, deleting, and inspecting **Agent Health Monitoring (AHM)** data alerts, and for verifying the metric values behind them. This is the same functionality the AHM Setup UI performs at `/lightning/n/standard-AgentforceStudio?c__nav=alerts`, but driven entirely from the `sf` CLI.

Use this reference when the user wants to:

- Create / list / delete an AHM alert on an agent metric (escalation rate, deflection rate, etc.)
- Understand why an alert isn't firing
- Inspect triggered alert notifications (trigger history)
- Discover the alertable SDMs, metrics, and agent filter values on an org

---

## Contents

1. [CLI-first — how commands are expressed here](#cli-first--how-commands-are-expressed-here)
2. [The endpoint: `tableau/dataAlerts`](#the-endpoint-tableaudataalerts)
3. [List and describe alerts](#list-and-describe-alerts)
4. [Prerequisite: confirm the three alertable SDMs](#prerequisite-for-creating-alerts-confirm-the-three-alertable-sdms)
5. [Enumerate `_mtc` metrics and screen for time-grain fit](#enumerate-_mtc-metrics-and-screen-for-time-grain-fit)
6. [List the agents available for alert filters](#list-the-agents-available-for-alert-filters)
7. [Create an alert](#create-an-alert) — POST vs GET field names, `filterContext`, thresholds, full schema, enums, troubleshooting
8. [Update an alert](#update-an-alert) — PUT in place
9. [Delete an alert](#delete-an-alert)
10. [Trigger history — inspect triggered notifications](#trigger-history--inspect-triggered-notifications)
11. [Verify metric value via Semantic Engine Gateway](#verify-metric-value-via-semantic-engine-gateway-condensed)
12. [Global troubleshooting](#global-troubleshooting)

---

## CLI-first — how commands are expressed here

Every operation is expressed through **`sf api request rest`**, which authenticates and executes with the org credentials the CLI already holds (`-o <org>`).

```bash
# Generic shape used throughout this reference
sf api request rest "<path>" -o <org>                          # GET
sf api request rest "<path>" -X POST -H "Content-Type: application/json" -b "@$body" -o <org>
sf api request rest "<path>/<id>" -X DELETE -o <org>
```

`sf api request rest` supports `-X/--method` (GET|POST|PUT|PATCH|DELETE|…), `-H/--header`, and `-b/--body` (`@file` to read a file, `-` for stdin, `""` for empty). Pipe the output to `python3 -m json.tool` to pretty-print.

> **Request bodies go in a private temp file — never a fixed `/tmp/<name>.json`.** The POST bodies below carry org data (SDM/agent names, user IDs, alert config). A predictable, world-readable path in the shared `/tmp` namespace invites info disclosure, symlink clobbering, and a TOCTOU swap between the `cat >` write and the `-b @file` read. Create a per-invocation `0600` file, pass it, and remove it on exit:
>
> ```bash
> body=$(mktemp) && chmod 600 "$body"
> trap 'rm -f "$body"' EXIT
> # cat > "$body" <<'JSON' … then: -b "@$body"
> ```
>
> Every heredoc example below assumes this `body=$(mktemp)` idiom (reuse the same `trap` for the run) and writes to `"$body"` rather than a literal `/tmp/*.json`.

> **Notes.**
> - **The org must already be authenticated — these commands do not launch a login.** Every call uses the credentials the CLI already holds for `-o <org>`. Pre-flight with `sf org display --target-org <org>` (or `sf org list`); if the alias is missing/unauthenticated, `sf` fails with `No authorization information found for <alias>` rather than prompting. Authenticate with `sf org login web --alias <alias>` — a browser flow the **user** runs (an agent can't complete it; in Claude Code prefix the command with `!`). An **expired** token auto-refreshes on the next call; a **missing/removed** auth does not.
> - There is **no dedicated `sf agent alert …` subcommand today** — the generic `sf api request rest` wrapper is the CLI surface for AHM alerts. If/when dedicated subcommands ship, prefer them and their `--help`.
> - `sf api request rest` is GA and bundled with the Salesforce CLI (no extra plugin required); it is the sole API surface used throughout this reference.
> - **API version.** Every endpoint below uses **`/services/data/v66.0/`**. For these resources the data API version is largely orthogonal to resource availability — v66.0 is the standardized floor here and `v68.0` also works on current orgs. Only raise the version if you need a field added in a later release; if any call unexpectedly returns `404`, retry at the org's current API version before assuming the resource is missing.
> - Reuse **Phase 0 (Discover Data Space)** from the main skill for the `dataspace` value used by the Data Cloud SQL and semantic queries below (default: `default`).

### Resolve the owner user ID

The list endpoint requires an `ownerId`. Resolve the current user (or another user by username):

```bash
# Current user (preferred — no user-supplied input)
USER_ID=$(sf org display user --target-org <org> --json \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["id"])')

# Another user, supplied out-of-band as a validated Salesforce User Id (15 or 18 chars).
# Prefer an Id over a username — it needs no query and can't inject.
USER_ID="005XXXXXXXXXXXXXXX"
[[ "$USER_ID" =~ ^[a-zA-Z0-9]{15,18}$ ]] || { echo "Not a valid User Id" >&2; exit 1; }
```

**Do not** build the owner query by interpolating a free-text username into `--query "… Username = '<value>'"`. That value crosses two hostile boundaries at once — the SOQL string literal (`x' OR IsActive=true LIMIT 1` subverts the predicate → resolves an arbitrary user's Id) and the surrounding double-quoted shell word (`$(…)`/backticks execute before `sf` runs). `agentforce-observe` reads production session data, so a username can be attacker-influenced, not just operator-typed.

If a username lookup is genuinely unavoidable, first reject anything with quoting/shell metacharacters, then build the SOQL in a `python3` snippet that reads the value from an environment variable and escapes the literal — never inline in a double-quoted `--query`:

```bash
UNAME='other@example.com'
# Reject any quote / backtick / $ / backslash / ; before the value is used.
printf '%s' "$UNAME" | LC_ALL=C grep -q "[\"'\`\$\\;]" && { echo "Reject: unsafe chars in username" >&2; exit 1; }
# UNAME is already validated above (no quotes/backslashes/shell metachars), so interpolate it directly.
SOQL=$(UNAME="$UNAME" python3 -c \
  'import os;print("SELECT Id FROM User WHERE Username = \x27%s\x27" % os.environ["UNAME"])')
USER_ID=$(sf data query --target-org <org> --json --query "$SOQL" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["records"][0]["Id"])')
```

---

## The endpoint: `tableau/dataAlerts`

AHM alerts are a special flavor of Salesforce data alerts — `dataAlertType: "agenthealthmonitoring"` — exposed under a dedicated REST resource:

```text
/services/data/v66.0/tableau/dataAlerts
```

They are **not** served from `/wave/dataAlerts` (CRM Analytics) or `/analytics/dataAlerts`. The "tableau" namespace here is Tableau Next / Data Cloud analytics, distinct from classic Wave — `/tableau/dataAlerts` does **not** require Wave / CRM Analytics.

| Method | Path | Purpose                                     |
|--------|------|---------------------------------------------|
| GET    | `/services/data/v66.0/tableau/dataAlerts?ownerId={userId}` | List alerts for a user (requires `ownerId`) |
| POST   | `/services/data/v66.0/tableau/dataAlerts` | Create                                      |
| PUT    | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Update (same input representation as POST)  |
| DELETE | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Delete                                      |
| GET    | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Not supported — 405 Method Not Allowed      |

Single-alert GET is not supported — list all and filter client-side. Omitting `ownerId` on the list GET returns `400 MISSING_PARAM: Owner ID cannot be empty`. There is no "list all alerts across the org" shape; always scope to a user.

---

## List and describe alerts

```bash
sf api request rest "/services/data/v66.0/tableau/dataAlerts?ownerId=$USER_ID" -o <org> \
  | python3 -m json.tool
```

Example response (`totalSize: 1`):

```json
{
  "dataAlerts": [
    {
      "id": "3VRxx0000001xxxxAY",
      "alertName": "te1_AHM_High_AHM_Escalation_Rate_mtc::Service_Agent_Analytics_SDM_1f8_AHM_Above_0.1_AHM_All",
      "dataAlertType": "agenthealthmonitoring",
      "createdDate": "2026-05-01T23:01:02.000Z",
      "schedule": {"type": "everynminutes", "minuteLevelFrequency": 1},
      "thresholds": {
        "conditions": [{
          "leftOperand": {
            "type": "insights",
            "factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
            "filterContext": [],
            "insightType": "popc",
            "metricId": "1HUxx000000xxxx4AE",
            "modelApiNameOrId": "2SMxx000000xxxx4AY",
            "timeContext": {"operator": "LastNHours", "values": ["1"]}
          },
          "operator": "greaterorequal",
          "rightOperand": {"type": "rawvalue", "dataType": "number", "value": "0.1"}
        }],
        "customLogicalOperation": "1"
      },
      "deliveryConfigurations": {
        "receivers": [
          {"type": "notification", "recipients": ["005xx000001xxxxAAC"]},
          {"type": "email", "recipients": ["005xx000001xxxxAAC"]}
        ]
      }
    }
  ],
  "totalSize": 1
}
```

Observations from a live response:

1. **`alertName` is an encoded form** — `_AHM_`-separated: `<freeText>_AHM_<severity>_AHM_<metricApiName>::<sdmApiName>_AHM_<operator>_<threshold>_AHM_<scope>`.
2. **`modelApiNameOrId` can be an ID** (e.g. `2SMxx…`), not always an API name.
3. **`filterContext: []`** even when the alert name indicates filters — this is a **known bug**. The real filters live in the **sub-metric** referenced by `metricId`.
4. **`schedule.minuteLevelFrequency: 1`** — fires every minute (aggressive; useful for test/demo).

### Retrieve the sub-metric to see the actual filters

```bash
METRIC_ID="1HUxx000000xxxx4AE"
MODEL="Service_Agent_Analytics_SDM"   # or the ID from modelApiNameOrId
sf api request rest \
  "/services/data/v66.0/ssot/semantic/models/${MODEL}/sub-metrics/${METRIC_ID}" -o <org> \
  | python3 -m json.tool
```

HTTP 200 → it's a filtered sub-metric; `filters[]` holds the real agent-name / agent-type conditions.

---

## Prerequisite for creating alerts: confirm the three alertable SDMs

> **Scope rule.** AHM data alerts target **exactly three** semantic data models. **Ignore every other SDM** returned by the endpoint — including `Agent_Health_Monitoring_SDM`, dataset-specific models, or custom models. If the metric you want isn't in one of the three below, the answer is "not supported," not "try another SDM."

1. **Agentforce Analytics Foundations** — base model. API name `sfm_Agentforce_Analytics_Foundations` (stable). Exposes only `_clc`, no `_mtc`.
2. **Employee Agent Analytics SDM** — match by **label** `Employee Agent Analytics SDM` or **`app` = `Employee_Agent_Analytics`**.
3. **Service Agent Analytics SDM** — match by **label** `Service Agent Analytics SDM` or **`app` = `Service_Agent_Analytics`**.

> **Never hard-code the `_1f8` (or any other) `apiName` suffix.** It's a content-hash / provisioning token that changes when the app template is re-provisioned or moved to another org. Always discover the live `apiName` by listing models and matching on the stable fields (`label`, `app`), then read `apiName` back out.

### List all semantic models

```bash
sf api request rest "/services/data/v66.0/ssot/semantic/models" -o <org> \
  | python3 -m json.tool
```

Example `items[]` entry:

```json
{
  "apiName": "Service_Agent_Analytics_SDM_1f8",
  "app": "Service_Agent_Analytics",
  "label": "Service Agent Analytics SDM",
  "id": "2SMxx000000xxxx4AY",
  "dataspace": "default",
  "baseModels": [
    {"apiName": "sfm_Agentforce_Analytics_Foundations", "label": "Agentforce Analytics Foundations"}
  ]
}
```

Key fields: `label` (stable match), `apiName` (goes into `modelApiNameOrId`, has the suffix), `app`/`sourceCreationName` (stable app identifiers), `id` (SDM record ID — needed for the semantic-engine gateway).

### Provisioning check — the three only, ignore everything else

```bash
export ORG=<org>
sf api request rest "/services/data/v66.0/ssot/semantic/models" -o "$ORG" | python3 <<'PY'
import json, sys
models = json.load(sys.stdin).get("items", [])

# Allowlist: exactly the three AHM-alertable SDMs. Match by stable fields only.
IN_SCOPE = [
    ("Agentforce Analytics Foundations", None),
    ("Employee Agent Analytics SDM",     "Employee_Agent_Analytics"),
    ("Service Agent Analytics SDM",      "Service_Agent_Analytics"),
]

def match(m, label, app):
    return m.get("label") == label or (app and m.get("app") == app)

print(f"{'SDM (in scope)':40} {'apiName':45} {'id':25} status")
print("-" * 120)
resolved_ids = set()
for label, app in IN_SCOPE:
    hits = [m for m in models if match(m, label, app)]
    if not hits:
        print(f"{label:40} {'(not found)':45} {'':25} Not provisioned — cannot alert on this SDM")
        continue
    m = hits[0]
    resolved_ids.add(m.get("id"))
    print(f"{label:40} {m.get('apiName','?'):45} {str(m.get('id','?')):25} provisioned")

ignored = [m for m in models if m.get("id") not in resolved_ids]
print(f"\nIgnored — out of scope ({len(ignored)} model(s)). Do NOT alert on these:")
for m in ignored:
    print(f"  • {m.get('label','?')} (apiName={m.get('apiName','?')})")
if not ignored:
    print("  (none)")
PY
```

Interpreting the output:
- All three provisioned → proceed to create alerts; use the `apiName` values as-is for `modelApiNameOrId`.
- Any Not provisioned → that SDM's app template is not provisioned on this org. Alerts cannot be created against it until the template is deployed (a provisioning task, not a CLI workaround).
- Anything "Ignored" → acknowledge and move on. **Do not** substitute an ignored SDM for a missing in-scope one.

Troubleshooting:
- **Only Foundations shows up** — the Employee/Service app templates haven't been provisioned; they must be deployed via the Analytics app template flow first.
- **`FUNCTIONALITY_NOT_ENABLED` on `/ssot/semantic/models`** — Data Cloud / SSOT is not provisioned; none of the alertable SDMs can exist.
- **API name doesn't match `modelApiNameOrId` in an existing alert** — the alert was created against an older provisioning hash. Delete and recreate against the current `apiName`.

---

## Enumerate `_mtc` metrics and screen for time-grain fit

The alertable atoms inside each SDM are **semantic metrics** whose API names end in `_mtc`. These differ from the `_clc` calculated measurements:

| Endpoint | Object | Suffix | What it is |
|---|---|---|---|
| `…/calculated-measurements` | Calculated measurement | `_clc` | Low-level aggregation at the SDM layer (raw ingredient). |
| `…/metrics` | Semantic metric | `_mtc` | Wraps a `_clc` with time dimension, grains, insight settings. **This is what data alerts reference.** |
| `…/sub-metrics` | Sub-metric | id `1HU…` | A filtered instantiation of a `_mtc`. Most real alerts' `metricId` is a sub-metric, not the bare `_mtc`. |

### List all `_mtc` across the three SDMs

```bash
export ORG=<org>
python3 <<'PY'
import json, subprocess

def rest(path):
    out = subprocess.check_output(["sf","api","request","rest",path,"-o",__import__("os").environ["ORG"]])
    return json.loads(out)

models = rest("/services/data/v66.0/ssot/semantic/models").get("items", [])
IN_SCOPE_APPS = {
    "Agentforce Analytics Foundations": None,
    "Employee Agent Analytics SDM":     "Employee_Agent_Analytics",
    "Service Agent Analytics SDM":      "Service_Agent_Analytics",
}
resolved = {}
for m in models:
    for label, app in IN_SCOPE_APPS.items():
        if m.get("label") == label or (app and m.get("app") == app):
            resolved[label] = m["apiName"]
            break

for label, api in resolved.items():
    metrics = rest(f"/services/data/v66.0/ssot/semantic/models/{api}/metrics").get("metrics", [])
    mtc = [m for m in metrics if m.get("apiName","").endswith("_mtc")]
    print(f"\n=== {label} ({api})  —  {len(mtc)} _mtc metric(s) ===")
    for m in mtc:
        grains = ",".join(m.get("timeGrains", [])) or "(no timeGrains)"
        print(f"  {m['apiName']:40} | {m.get('label',''):30} | id={m.get('id','')} | grains=[{grains}]")
    if not mtc:
        print("  (none — e.g. Foundations exposes only _clc)")
PY
```

On a typical org, Foundations exposes **0** `_mtc`; Service Analytics SDM exposes ~14 (`Escalation_Rate_mtc`, `Deflection_Rate_mtc`, `Abandonment_Rate_mtc`, `Engagement_Rate_mtc`, `Agent_Health_Score_mtc`, `Total_Sessions_mtc`, …); Employee Analytics SDM exposes ~9 (`Stickiness_Rate_mtc`, `Total_Unique_Users_mtc`, `Weekly_Active_Users_mtc`, …). All declare `timeGrains = [Day, Week, Month, Quarter, Year]` — none declare Hour/Minute, but the `dataAlerts` API doesn't enforce this (the alert's `timeContext.operator` is independent).

Pull the `id` of the chosen `_mtc` — it goes into `thresholds.conditions[].leftOperand.metricId` when creating an alert.

### Screening: does the metric make sense at a short window?

The SDM doesn't block nonsense; apply judgement:

| Metric shape | Smallest useful window | Short-window (≤ 1h) alert? |
|---|---|---|
| **Ratios / rates** — Engagement, Deflection, Abandonment, Escalation, Stickiness | 15 min – 1 h | Yes — self-normalizing. Add a minimum-volume guard. |
| **Averages / times** — Avg Time to Deflection/Escalation/Execution, Avg Quality Score | ≥ 1 h | Borderline at 1–4h; daily is fine. Not at 15 min. |
| **Cumulative totals** — Total/Deflected/Escalated/Abandoned/Engaged Sessions, Total Actions | ≥ 1 h, typically ≥ 1 day | No — noisy at short windows — alert on the corresponding **rate** instead. |
| **Cardinality counts** — Unique Users, Weekly Active Users, Total Employee Agents | ≥ 1 day | No — smallest sensible window Day (DAU) / Week (WAU). |
| **Composite scores** — Agent Health Score | default ≥ 1 day | Depends on inputs; default no. |

> **Minimum-volume guard.** Even for ratios, check the denominator: a 100% error rate on 1 session in 15 min is noise. The API has no built-in guard — pre-filter via a sub-metric requiring N > threshold sessions, or combine a rate threshold AND an absolute-count threshold in `thresholds.conditions[]`.

Use short-window rate metrics with `schedule.minuteLevelFrequency ≤ 60` and `timeContext.operator = LastNHours|LastNMinutes`; use daily+ metrics only with `minuteLevelFrequency ≥ 1440` and `LastNDays`/`LastNWeeks`.

---

## List the agents available for alert filters

The UI Create-Alert modal scopes an alert to an **Agent Name** or **Agent Type**. Those values come from **live session-tracing data (Data Cloud DLO)**, not from agent metadata.

> **For AHM alerts, use the DLO query (ground truth) below.** If a name appears in `BotDefinition` but has never run a session, the filter selects zero sessions and the alert never fires. A name in the DLO without a `BotDefinition` (demo/external agents) is still alertable — the filter matches strings, not IDs.
>
> **Cross-link:** the DLO object below, `ssot__AiAgentSessionParticipant__dlm`, is the same DMO documented in [`stdm-schema.md`](stdm-schema.md). Note the AHM filter uses `ssot__AiAgentApiName__c` here, which is **not** the same as the `GenAiPlannerDefinition` `MasterLabel`/`DeveloperName` the main skill resolves for STDM `findSessions` — grab the filter value from this query, don't reconstruct it.

### DLO query (ground truth)

Write the SQL body to a private temp file (see the `mktemp` idiom above) to avoid shell-escaping, then POST it:

```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
  "sql": "SELECT ssot__AiAgentApiName__c AS agent_name, ssot__AiAgentType__c AS agent_type, COUNT(*) AS participant_rows FROM ssot__AiAgentSessionParticipant__dlm WHERE ssot__AiAgentSessionParticipantRole__c != 'USER' GROUP BY ssot__AiAgentApiName__c, ssot__AiAgentType__c ORDER BY agent_name",
  "rowLimit": 500,
  "adaptiveTimeout": 1
}
JSON

sf api request rest "/services/data/v66.0/ssot/query-sql?dataspace=default" \
  -X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
  | python3 -m json.tool
```

The `ssot__AiAgentSessionParticipantRole__c != 'USER'` clause filters out the human side of the conversation. `GROUP BY` already yields one row per distinct `(agent_name, agent_type)` pair — no `DISTINCT` needed. In the `data` array: column 0 = `agent_name` (→ `AI_Agent_Api_Name` filter values), column 1 = `agent_type`, column 2 = `participant_rows` (volume sanity check — an agent with a handful of rows is too sparse for a 15-min alert).

### Metadata cross-reference (fallback)

```bash
sf data query --target-org <org> \
  --query "SELECT Id, DeveloperName, MasterLabel, AgentType FROM BotDefinition ORDER BY DeveloperName"
```

Not required for alerting. `AiAgent` sObject is often not queryable via REST — don't rely on it.

### Next-Gen Authoring Bundles API (what agents are defined)

Works even where `BotDefinition`/`AiAgent` sObjects are not queryable (common on demo orgs):

```bash
sf api request rest "/services/data/v66.0/nextgen-authoring/bundles" -o <org> | python3 -m json.tool
# names only:
sf api request rest "/services/data/v66.0/nextgen-authoring/bundles" -o <org> \
  | python3 -c 'import sys,json; [print(b.get("apiName","?")) for b in json.load(sys.stdin).get("bundles", [])]'
```

`apiName` is the agent's developer name (the value usable as an alert filter); `isLegacy: false` = new Next-Gen agent. Requires `NextGenAuthoring.orgHasNextGenAgentAuthoringEnabled` + `…userCanAccessNextGenAgentAuthoring`, else 403 (the `.get("bundles", [])` above then prints nothing rather than raising).

**Use the DLO query (ground truth) for filter values.** The metadata cross-reference and Bundles API are for discovery/cross-ref only. An agent listed there but absent from the DLO has never run a session — alerting on it matches zero rows until sessions arrive.

Troubleshooting:
- **`query-sql` 400 / FUNCTIONALITY_NOT_ENABLED** — SSOT/Data Cloud not enabled; no agent-tracing data exists and alerts can't fire.
- **Zero rows** — no sessions have landed. Confirm session-tracing is on and data has flowed.
- **UI dropdown shows values not in the DLO query** — check the `dataspace`; change `?dataspace=` if the alert is scoped to a non-`default` space.

---

## Create an alert

`POST /services/data/v66.0/tableau/dataAlerts`. Inputs come from the sections above: `modelApiNameOrId` (SDM `apiName`/`id`), `metricId` (`_mtc` `id`), and `filterContext` (agent name/type from the DLO query above).

### POST field names ≠ GET field names

The POST input representation uses **different field names and casing** from the GET response:

| POST (input) | GET (output) | Notes |
|---|---|---|
| `utterance` | `alertName` | POST field is `utterance`; response returns `alertName` |
| `type: "Metric"` (leftOperand) | `type: "insights"` | PascalCase in, lowercase out |
| `type: "RawValue"` (rightOperand) | `type: "rawvalue"` | same pattern |
| `type: "EveryNMinutes"` (schedule) | `type: "everynminutes"` | same |
| `type: "Notification"` / `"Email"` | `type: "notification"` / `"email"` | same |
| `operator: "GreaterOrEqual"` | `operator: "greaterorequal"` | same |
| `insightType: "Popc"` | `insightType: "popc"` | same |

**Rule of thumb:** all `type` discriminators and enum values in the POST body use **PascalCase**; the GET response lowercases everything. Copying a GET response and POSTing it back fails with `JSON_PARSER_ERROR`.

### filterContext field-name format

Filter field names use **dot notation**: `{tableApiName}.{fieldApiName}`. The metric's `additionalDimensions` lists the available dimensions as `tableFieldReference` objects — concatenate `tableApiName` + `.` + `fieldApiName`:

```text
tableApiName: "Agent_API_Name_lv"
fieldApiName: "AI_Agent_Session_Participant2_AI_Agent_Api_Name"
→ filterContext fieldName: "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name"
```

Using just the `fieldApiName` fails with `INTERNAL_ERROR: Validation Failed: Invalid calculated Field`.

### Sub-metric auto-creation

When you POST with a non-empty `filterContext`, the backend **auto-creates a sub-metric** (ID prefix `1HU…`) from the bare `_mtc` id plus your filters. The response's `metricId` is the new sub-metric id, not the `_mtc` id you sent. You don't create sub-metrics manually. (The response's `filterContext` comes back `[]` — the known bug; fetch the sub-metric to see the real filters.)

### Threshold values are raw ratios, not display percentages

Rate metrics are a **0–1 ratio** (the UI multiplies by 100 for display; the alert threshold operates on the raw value):

| You want to alert at | Threshold `value` |
|---|---|
| 1% | `"0.01"` |
| 5% | `"0.05"` |
| 50% | `"0.5"` |
| 100% | `"1"` |

> `value: "1"` means ≥ 100%, not ≥ 1%. An alert with `"1"` on a rate metric is either a test alert designed to always fire, or a misconfiguration.

### Full POST schema

```json
{
  "utterance": "<alert name — encoded or freeform string>",
  "dataAlertType": "agenthealthmonitoring",
  "schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": <integer minutes> },
  "content": { "type": "Metric", "modelApiNameOrId": ["<SDM ID or apiName>"] },
  "thresholds": {
    "conditions": [
      {
        "leftOperand": {
          "type": "Metric",
          "modelApiNameOrId": "<SDM ID or apiName>",
          "metricId": "<_mtc id>",
          "insightType": "Popc",
          "factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
          "params": {},
          "filterContext": [
            { "fieldName": "<table.field dot notation>", "operator": "Equals", "values": ["<filter value>"] }
          ],
          "timeContext": { "operator": "LastNHours", "values": ["<hours>"] }
        },
        "operator": "GreaterOrEqual",
        "rightOperand": { "type": "RawValue", "dataType": "Number", "value": "<threshold as string>" }
      }
    ],
    "customLogicalOperation": "1"
  },
  "deliveryConfigurations": {
    "receivers": [
      { "type": "Notification", "recipients": ["<userId>"] },
      { "type": "Email", "recipients": ["<userId>"] }
    ]
  }
}
```

### Example: Escalation Rate ≥ 1% for agent `te1`, checked every 15 minutes

Build the body in a private temp file (see the `mktemp` idiom above), then POST it. The literal-value heredoc below shows the exact shape (verified against a real org) with **constant** example values — safe to paste as-is because nothing is substituted:

```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
  "utterance": "te1_AHM_High_AHM_Escalation_Rate_mtc::Service_Agent_Analytics_SDM_1f8_AHM_Above_0.01_AHM_te1",
  "dataAlertType": "agenthealthmonitoring",
  "schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": 15 },
  "content": { "type": "Metric", "modelApiNameOrId": ["2SMSG000000a6TV4AY"] },
  "thresholds": {
    "conditions": [
      {
        "leftOperand": {
          "type": "Metric",
          "modelApiNameOrId": "2SMSG000000a6TV4AY",
          "metricId": "1DOSG0000051KAm4AM",
          "insightType": "Popc",
          "factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
          "params": {},
          "filterContext": [
            { "fieldName": "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name", "operator": "Equals", "values": ["te1"] }
          ],
          "timeContext": { "operator": "LastNHours", "values": ["1"] }
        },
        "operator": "GreaterOrEqual",
        "rightOperand": { "type": "RawValue", "dataType": "Number", "value": "0.01" }
      }
    ],
    "customLogicalOperation": "1"
  },
  "deliveryConfigurations": {
    "receivers": [
      { "type": "Notification", "recipients": ["<USER_ID>"] },
      { "type": "Email", "recipients": ["<USER_ID>"] }
    ]
  }
}
JSON

sf api request rest "/services/data/v66.0/tableau/dataAlerts" \
  -X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
  | python3 -m json.tool
```

**When the agent name, `utterance`, filter value, threshold, or recipient come from data — not typed by hand — do not string-substitute them into the heredoc.** An agent name sourced from live DLO/session data can contain a `"` that breaks out of the JSON string and injects sibling fields (e.g. an attacker-controlled `Email` receiver that exfiltrates notifications, or a corrupted threshold that silently disables the alert). Reject values containing `"`/`\`, then let a JSON-aware builder encode everything — the body is byte-safe regardless of what the values contain:

```bash
# AGENT_NAME / UTTERANCE / THRESHOLD (raw ratio) / MODEL_ID / METRIC_ID / USER_ID come from earlier discovery.
case "$AGENT_NAME$UTTERANCE" in *[\"\\]*) echo "Reject: quote/backslash in name or utterance" >&2; exit 1;; esac
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
AGENT_NAME="$AGENT_NAME" UTTERANCE="$UTTERANCE" THRESHOLD="$THRESHOLD" \
MODEL_ID="$MODEL_ID" METRIC_ID="$METRIC_ID" USER_ID="$USER_ID" python3 - > "$body" <<'PY'
import os, sys, json
e = os.environ
json.dump({
  "utterance": e["UTTERANCE"],
  "dataAlertType": "agenthealthmonitoring",
  "schedule": {"type": "EveryNMinutes", "minuteLevelFrequency": 15},
  "content": {"type": "Metric", "modelApiNameOrId": [e["MODEL_ID"]]},
  "thresholds": {"conditions": [{
    "leftOperand": {"type": "Metric", "modelApiNameOrId": e["MODEL_ID"], "metricId": e["METRIC_ID"],
      "insightType": "Popc", "factKey": "FACT_KEY_TARGET_PERIOD_VALUE", "params": {},
      "filterContext": [{"fieldName": "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name",
        "operator": "Equals", "values": [e["AGENT_NAME"]]}],
      "timeContext": {"operator": "LastNHours", "values": ["1"]}},
    "operator": "GreaterOrEqual",
    "rightOperand": {"type": "RawValue", "dataType": "Number", "value": e["THRESHOLD"]}}],
    "customLogicalOperation": "1"},
  "deliveryConfigurations": {"receivers": [
    {"type": "Notification", "recipients": [e["USER_ID"]]},
    {"type": "Email", "recipients": [e["USER_ID"]]}]},
}, sys.stdout)
PY

sf api request rest "/services/data/v66.0/tableau/dataAlerts" \
  -X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
  | python3 -m json.tool
```

The response (HTTP 200) returns `id` (the alert id, prefix `3VR…`), and a `metricId` with prefix `1HU…` — the auto-created sub-metric, not the `1DO…` bare metric you sent. Response `filterContext` is `[]` (known bug); confirm the real filter via the sub-metric fetch shown earlier. All enum values in the response are lowercase.

### Example: unfiltered alert (all agents)

Same body, but `"filterContext": []` and (typically) `minuteLevelFrequency: 60` — no sub-metric is created; the alert references the bare `_mtc` id directly.

### Available enum values

- **Schedule types:** `"EveryNMinutes"` (+`minuteLevelFrequency`), `"Daily"` (+`hoursOfDay`), `"Weekly"` (+`hoursOfDay`,`daysOfWeek`).
- **Condition operators:** `"Equals"`, `"NotEqual"`, `"GreaterThan"`, `"GreaterOrEqual"`, `"LessThan"`, `"LessOrEqual"`.
- **Value types:** `"Metric"` (leftOperand); `"RawValue"` (rightOperand) with `dataType` `"Number"`/`"Text"`/`"Percent"`.
- **Receiver types:** `"Notification"` / `"Email"` (recipients = user IDs); `"Slack"` (recipients = Slack channel IDs).
- **Insight types:** `"Popc"` (period-over-period — used by AHM), `"RiskyMonopoly"`, `"Unspecified"`.
- **Filter operators:** `"Equals"`, `"DoesNotEquals"`, `"LessThan"`, `"GreaterThan"`, `"LessOrEqual"`, `"GreaterOrEqual"`, `"Between"`, `"IsNull"`, `"IsNotNull"`, `"Contains"`, `"DoesNotContain"`, `"StartsWith"`, `"EndsWith"`, `"In"`, `"NotIn"`.
- **Time-context operators:** `"LastNHours"`, `"LastNDays"`, `"LastNWeeks"`, `"LastNMonths"`, `"LastNQuarters"`, `"LastNYears"`, `"LastNMinutes"`.

### Constructing the `utterance` (alert name)

Conventional encoded format the AHM UI parses (the backend accepts arbitrary strings):

```text
{freeText}_AHM_{severity}_AHM_{metricApiName}::{sdmApiName}_AHM_{operator}_{threshold}_AHM_{scope}
```

`severity` = `High`/`Medium`/`Low`; `operator` = `Above`/`Below`; `scope` = `All` (no filter) or the filter value. Using this format lets the AHM UI display the alert correctly.

### Troubleshooting

- **`JSON_PARSER_ERROR: Unrecognized field`** — using a GET-response field name in the POST. Common: `alertName` → use `utterance`; `intervalInMinutes` → `minuteLevelFrequency`.
- **`JSON_PARSER_ERROR: Could not resolve type id`** — wrong casing on a `type` discriminator. Use PascalCase (`"Metric"`, `"RawValue"`, `"EveryNMinutes"`, …).
- **`INTERNAL_ERROR: Validation Failed: Invalid calculated Field`** — filter `fieldName` missing the table prefix; use dot notation.
- **`400` with no clear message** — `content.modelApiNameOrId` must be an **array** (`["2SM…"]`), not a string.
- **Alert created but never fires** — the agent name in `filterContext.values` must exactly match a DLO value (from the DLO query in "List the agents available for alert filters"). If the agent has never had a session, there's no data to alert on.

---

## Update an alert

`PUT /services/data/v66.0/tableau/dataAlerts/{alertId}` updates an alert **in place** — same `id`, no delete + recreate. It takes the **same input representation as POST** (PascalCase `type` discriminators, `utterance` not `alertName`, `RawValue`/`Metric`/`EveryNMinutes`, threshold as a raw-ratio string, etc. — see [Create an alert](#create-an-alert)). Send the **full** alert body, not a partial patch: the representation is replaced, so include every field you want to keep (schedule, content, all `thresholds.conditions`, `deliveryConfigurations`). Returns **HTTP 200** with the updated alert in GET-shape (lowercased enums); `id` is unchanged, `lastModifiedDate` is bumped.

> **Addressed by alert id.** Like `DELETE`, `PUT` targets the alert via its `{alertId}` in the **path** (`/tableau/dataAlerts/{alertId}`) — the id is *not* in the body. A metricId (`1HU…`) in the path 404s; use the alert id (`3VR…`).
>
> **Read-modify-write.** Fetch the current alert first (`GET tableau/dataAlerts?ownerId=$USER_ID`, filter by `id`), change only the fields you intend to, and PUT the whole thing back — translating the GET response's field names/casing to the POST input names (`alertName`→`utterance`, lowercase `type`s→PascalCase). Keep the existing `metricId` in the body: PUT preserves it (no sub-metric churn) as long as `filterContext` is unchanged.

### Example: change a threshold (e.g. `80` → `85`)

Build the full body in a private temp file (see the `mktemp` idiom above), then PUT it. This mirrors the POST schema with the alert's current values, changing only `rightOperand.value` (and, optionally, `utterance`):

```bash
ALERT_ID="3VRSG0000001uBR4AY"
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
  "utterance": "Task Resolution Rate Less Than 85% for AgentRuntimeInsights262",
  "dataAlertType": "agenthealthmonitoring",
  "schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": 10 },
  "content": { "type": "Metric", "modelApiNameOrId": ["2SMSG000000bPXf4AM"] },
  "thresholds": {
    "conditions": [
      {
        "leftOperand": {
          "type": "Metric",
          "modelApiNameOrId": "2SMSG000000bPXf4AM",
          "metricId": "1HUSG0000010Un74AE",
          "insightType": "Popc",
          "factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
          "params": {},
          "filterContext": [],
          "timeContext": { "operator": "LastNMinutes", "values": ["15"] }
        },
        "operator": "LessThan",
        "rightOperand": { "type": "RawValue", "dataType": "Number", "value": "85" }
      }
    ],
    "customLogicalOperation": "1"
  },
  "deliveryConfigurations": {
    "receivers": [
      { "type": "Notification", "recipients": ["005SG00000ZEQePYAX"] },
      { "type": "Email", "recipients": ["005SG00000ZEQePYAX"] }
    ]
  }
}
JSON

sf api request rest "/services/data/v66.0/tableau/dataAlerts/$ALERT_ID" \
  -X PUT -H "Content-Type: application/json" -b "@$body" -o <org> \
  | python3 -m json.tool
```

When the new name/threshold/recipients come from **data** rather than being typed by hand, do not string-substitute into the heredoc — reject values containing `"`/`\` then build the JSON with the same `python3` JSON-dump idiom shown under [Create an alert](#create-an-alert) (the injection risk is identical for PUT).

**Verified (org `ahm-test-notifications`, 2026-09-02):** PUT on alert `3VRSG0000001uBR4AY` with the POST-style body above returned HTTP 200, changed `rightOperand.value` `80`→`85` and `alertName`, kept the same `id` and `metricId` (`1HUSG0000010Un74AE`), and bumped `lastModifiedDate` — a true in-place update. Reverting to `80` via a second PUT worked identically.

### Troubleshooting

- **Same `JSON_PARSER_ERROR` / casing / array-shape errors as POST** — the input grammar is identical; see [Create an alert → Troubleshooting](#troubleshooting).
- **`404` on the PUT** — confirm the `{alertId}` is a real alert `id` (`3VR…`) from the list; a metricId (`1HU…`) will 404. If the id is valid and it still 404s, retry at the org's current API version.
- **Fields silently reverting** — PUT replaces the representation; a field you omitted reverts to server default. Always send the full body (read-modify-write), not a partial patch.

---

## Delete an alert

```bash
ALERT_ID="3VRSG0000001kX34AI"   # from the list or the create response
sf api request rest "/services/data/v66.0/tableau/dataAlerts/$ALERT_ID" -X DELETE -o <org> --include
```

Returns **HTTP 204** (No Content) on success — no response body. Use `--include` to see the status line.

---

## Trigger history — inspect triggered notifications

When an alert's condition is met it generates a **system notification** and optionally an **email** (per `deliveryConfigurations.receivers`). AHM alert notifications carry the notification `type` **`templatized_data_alert`** and their `target` field holds the **sub-metric id** (`1HU…`) — the same value as the alert's `thresholds.conditions[0].leftOperand.metricId`.

**Attribute by alert id only.** The notification's `targetPageRef` (an HTML-encoded JSON string) embeds the owning alert id under `state.c__alertId` (or `state.c__dataAlertId`, or nested `attributes.pageRef.state.c__alertId`/`c__dataAlertId`). That id is the **precise** attribution key — match on it. **Do not fall back to the `target == metricId` join**: it is only correct when there's one alert per metric — two alerts on the same bare `_mtc` (both with empty `filterContext`) share a `metricId`, so the metricId join returns the **union** of their firings and cannot tell them apart. A notification whose alert id can't be recovered is simply **not attributed** (see the null-handling note below), not routed through metricId.

### Extracting and matching the alert id

```python
import json, html

def extract_alert_id(n):
    """Return the owning alert id from a notification, or None. Degrades to None
    (never raises) on missing / non-JSON / key-absent targetPageRef → 'no match'."""
    ref = n.get("targetPageRef")
    if not ref:
        return None
    try:
        if isinstance(ref, str):
            ref = json.loads(html.unescape(ref))   # API HTML-encodes the quotes (&quot;)
    except Exception:
        return None
    top = (ref or {}).get("state") or {}
    if top.get("c__dataAlertId"): return top["c__dataAlertId"]
    if top.get("c__alertId"):     return top["c__alertId"]
    nested = (((ref or {}).get("attributes") or {}).get("pageRef") or {}).get("state") or {}
    return nested.get("c__dataAlertId") or nested.get("c__alertId")

def ids_match(a, b):
    """15/18-char-safe Salesforce id equality."""
    if not a or not b:                       # None from a failed extraction ⇒ no match, drop
        return False
    return a == b or a[:15] == b[:15]         # 18-char = 15-char + 3-char checksum suffix
```

- **`None` extraction is a non-match, not an error.** A missing/malformed/key-absent `targetPageRef` yields `None`; `ids_match` returns `False` on a `None` operand, so the notification is simply **dropped** (not attributed to any alert) — nothing throws, and it is **not** routed through a metricId fallback.
- **Compare the first 15 chars.** The id embedded in `targetPageRef` is frequently the **15-char** form; the alert id from `GET tableau/dataAlerts` is the **18-char** form (15-char + 3-char case-insensitive checksum). Strict `===` fails on the same record (`3VRSG0000001vHB` vs `3VRSG0000001vHB4AY`); try exact first, then `a[:15] == b[:15]`.
- **Casing caveat.** The 15-char id is case-sensitive and the 18-char checksum encodes that casing; `ids_match` does **not** case-normalize, so truncating to 15 is correct only because both sides preserve the true casing (they do in this flow — neither source upper/lower-cases the id). Do not add a `.lower()` on either side.
- **Verified (org `ahm-test-notifications`):** all fired notifications carried `state.c__alertId` (18-char, e.g. `3VRSG0000001vHB4AY`) matching a listed alert 1:1; the nested branches and `c__dataAlertId` are defensive fallbacks for other page-ref shapes.

### The `X-UNS-Type-Filter: all` header is REQUIRED to see AHM notifications

The Connect Notifications API (`connect/notifications*`) **defaults to returning *custom* notification types only**. AHM/Tableau-Next alerts are **standard/platform** types (`templatized_data_alert`), so a plain call returns an empty list / empty types even when alerts have genuinely fired. **Always pass `-H "X-UNS-Type-Filter: all"`** on every `connect/notifications*` call to include standard/platform types:

```bash
# Status (counts). WITHOUT the header these are custom-types-only and read 0 for AHM.
sf api request rest "/services/data/v66.0/connect/notifications/status" \
  -H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool

# List all notifications (includes AHM). The header is what surfaces them.
sf api request rest "/services/data/v66.0/connect/notifications" \
  -H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool

# Which notification types the org exposes (verify templatized_data_alert is present).
sf api request rest "/services/data/v66.0/connect/notifications/types" \
  -H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool
```

> **Symptom of the missing header:** `connect/notifications` → `{ "notifications": [] }`, `connect/notifications/types` → `{ "notificationTypes": [] }`, and `status` → `unreadCount: 0` — even though the alerts fired and are visible in the UI bell. This empty result is a **false negative**, not proof of "not fired." (The UI bell uses the internal Aura controller `ui-notifications-components-notifications-controller.Notifications.getNotifications`, which reads the full store; the CLI must opt in with the header.)

Each notification object includes: `type`, `messageTitle` (metric + agent + current value), `messageBody` (`The current value (X) is above/below the alert condition (Y)`), `lastModified` (fire time), `read`/`seen`, `target` (the sub-metric id), and `targetPageRef` (an HTML-encoded JSON string). The **owning alert id** is in `targetPageRef.state.c__alertId` — that, not `target`, is the attribution key (see "Extracting and matching the alert id" above).

### Answering "get notifications" (default response)

When the user asks to get / list / check notifications for an org, report **both**:
1. **Status** — `unreadCount` / `unseenCount` from the `status` call (with the header).
2. **Notifications** — the list from the `notifications` call (with the header), summarized (title, body, fire time, type).

Filter to `type == "templatized_data_alert"` to isolate AHM/agent-health alerts (the type is shared with other templatized Tableau data alerts, so it is **not** AHM-exclusive — filter, don't assume).

### Answering "notifications for a specific alert"

There is **no per-alert notification endpoint** (`…/dataAlerts/{id}/notifications` → 404). Instead, fetch **all** notifications once, then **filter client-side by alert id** (exact — see "Extracting and matching the alert id" above):

1. Fetch all notifications with the header (above).
2. For each notification, `extract_alert_id(n)` and keep those where `ids_match(extracted, targetAlertId)` (15/18-char-safe). Those are the firings of that alert.
3. A notification whose `targetPageRef` is missing/unparseable (`extract_alert_id` → `None`) is **dropped**, not attributed. Do **not** fall back to a `target == metricId` join — it mis-attributes when two alerts share a metric.

Present title / current value / threshold / `lastModified` per firing.

```bash
# Attribute every notification to its owning alert (by alert id from targetPageRef — NOT metricId).
ORG=<org>
USER_ID=$(sf org display user --target-org "$ORG" --json | python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["id"])')
sf api request rest "/services/data/v66.0/tableau/dataAlerts?ownerId=$USER_ID" -o "$ORG" > /tmp/alerts.json
sf api request rest "/services/data/v66.0/connect/notifications" -H "X-UNS-Type-Filter: all" -o "$ORG" > /tmp/notifs.json
python3 - <<'PY'
import json, html
from collections import defaultdict

def extract_alert_id(n):
    ref = n.get("targetPageRef")
    if not ref:
        return None
    try:
        if isinstance(ref, str):
            ref = json.loads(html.unescape(ref))
    except Exception:
        return None
    top = (ref or {}).get("state") or {}
    if top.get("c__dataAlertId"): return top["c__dataAlertId"]
    if top.get("c__alertId"):     return top["c__alertId"]
    nested = (((ref or {}).get("attributes") or {}).get("pageRef") or {}).get("state") or {}
    return nested.get("c__dataAlertId") or nested.get("c__alertId")

def norm(x):
    return x[:15] if x else x   # 15/18-char-safe: truncate both sides to the 15-char id

alerts = json.load(open('/tmp/alerts.json')).get("dataAlerts", [])
notifs = json.load(open('/tmp/notifs.json')).get("notifications", [])
by_alert = {norm(a.get("id")): a for a in alerts}         # key on the alert id (15-char-normalized)
groups = defaultdict(list)
for n in notifs:
    groups[norm(extract_alert_id(n))].append(n)           # None-key bucket = no recoverable alert id
for aid, a in by_alert.items():
    ns = sorted(groups.get(aid, []), key=lambda x: x.get("lastModified", ""))
    print(f"\nALERT: {a.get('alertName')}  (id={a.get('id')}) -> {len(ns)} fired")
    for n in ns:
        print("   %s | %s" % (n.get("lastModified", "")[:19], html.unescape(n.get("messageTitle", ""))))
# notifications whose extracted alert id matches no listed alert (deleted/renamed), or None (no recoverable id — dropped, NOT routed to metricId)
for aid, ns in groups.items():
    if aid in by_alert:
        continue
    for n in ns:
        label = ("UNMATCHED alertId=%s" % aid) if aid else "UNATTRIBUTED (no alert id in targetPageRef)"
        print("%s | %s" % (label, html.unescape(n.get("messageTitle", ""))))
PY
```

### Other verification paths

- **UI (per-alert trigger history)** — `/lightning/n/standard-AgentforceStudio?c__nav=alerts` → **Incidents** tab shows the full, per-alert notification list with details. Use it to corroborate the CLI attribution above or when the header path is unavailable.
- **Email** — if `deliveryConfigurations` includes `"type": "Email"`, the owner receives an email when it fires.
- **Metric value** — confirm the underlying metric actually crosses the threshold (see next section).

### Troubleshooting alerts that don't fire

First rule out the **missing-header false negative**: if the list is empty, retry with `-H "X-UNS-Type-Filter: all"` before concluding the alert never fired. If it is still empty (no notification attributed to that alert id — see attribution above), then check:
1. **Does the SDM return data?** If the semantic gateway (below) returns null/0, the alert evaluation also returns null and the condition is never met.
2. **Is the schedule running?** Alerts evaluate on their `minuteLevelFrequency`; a freshly created alert may take a few cycles.
3. **Does the threshold make sense?** A 10% rate with threshold `>= 0.5` (50%) never fires — remember the raw-0–1-ratio scale.
4. **Does the filter match live data?** An agent name in `filterContext` that never had a session selects zero rows — nothing to alert on.

---

## Verify metric value via Semantic Engine Gateway (condensed)

Use this to confirm a metric actually crosses the threshold before concluding an alert is broken. The gateway queries any `_clc` calculated measurement with time grouping / filters / aggregation.

**Endpoint:** `POST /services/data/v66.0/semantic-engine/gateway`. Inputs: SDM **record ID** (the `id`, not `apiName`), the `_clc` name (from the metric's `measurementReference.calculatedFieldApiName`), the time dimension (`AI_Agent_Session.Start_Timestamp` for session-level rates), and an ISO date range. Rate values come back as **raw 0–1 ratios**, same scale as alert thresholds.

### Single aggregate value (e.g. overall Escalation Rate for the last 24h)

```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
  "structuredSemanticQuery": {
    "fields": [
      {
        "expression": { "semanticField": { "name": "Escalation_Rate_clc" } },
        "alias": "Escalation_Rate",
        "rowGrouping": false,
        "semanticAggregationMethod": "SEMANTIC_AGGREGATION_METHOD_USER_AGG"
      }
    ],
    "topNFilter": { "rowsNumber": 1, "sortOrders": [] },
    "options": { "limitOptions": { "limit": 2 }, "sortOrders": [], "grandTotal": false },
    "semantic_context": { "currency": { "id": "" } },
    "flattenFilter": {
      "filters": [
        { "fieldName": "AI_Agent_Session.Start_Timestamp", "operator": "Between", "value": "2026-05-02T00:00:00.000Z|2026-05-03T00:00:00.000Z" }
      ],
      "filterLogic": "1"
    }
  },
  "semanticModelId": "2SMSG000000a6TV4AY"
}
JSON

sf api request rest "/services/data/v66.0/semantic-engine/gateway" \
  -X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
  | python3 -m json.tool
```

### Time series (per-day) and agent filter

For a per-day series, add a first field with `"rowGrouping": true` and a `DATETRUNC('day', [AI_Agent_Session].[Start_Timestamp])` calculated field (alias e.g. `Day`), set `topNFilter.rowsNumber` = N, `options.limitOptions.limit` = N+1, and `grandTotal: true`. To scope to an agent, add a second `flattenFilter.filters` entry `{ "fieldName": "AI_Agent_Session_Participant.AI_Agent_Api_Name", "operator": "In", "value": "te1" }` (comma-separated for multiple) and set `"filterLogic": "1 AND 2"`.

Response rows carry `is_data_row__sl` (`true` = data row, `false` = grand total) and `grouping_1__sl` (`0` = detail, `1` = total); the `values` array is indexed by each field's `placeInOrder`.

### Discover available `_clc` measurements

```bash
SDM_API_NAME="Service_Agent_Analytics_SDM_1f8"
sf api request rest "/services/data/v66.0/ssot/semantic/models/${SDM_API_NAME}/calculated-measurements" -o <org> \
  | python3 -c 'import sys,json;[print(m["apiName"],"|",m.get("label",""),"|",m.get("dataType","")) for m in json.load(sys.stdin).get("items",[])]'
```

Common Service SDM measures: `Escalation_Rate_clc`, `Deflection_Rate_clc`, `Engagement_Rate_clc`, `Abandonment_Rate_clc`, `Error_Rate_clc`, `Unique_Sessions_clc`, `Escalated_Sessions_clc`, `Average_Session_Duration_clc`.

Troubleshooting:
- **`null` values** — verify data exists in the DLO (`SELECT COUNT(*) FROM ssot__AiAgentSession__dlm` via `query-sql`). DLO has rows but gateway returns null → the SDM-to-DLO mapping is broken (provisioning issue).
- **`USER_ILLEGAL_ARGUMENT_RESOLVE_ENTITY_ERROR: Failed to resolve semantic field`** — wrong `_clc` name; list measurements with the discovery command.
- **`404 Not Found`** — confirm the path is `/services/data/v66.0/semantic-engine/gateway`; if it still 404s, retry at the org's current API version (the resource may be exposed at a different version on that org).

---

## Global troubleshooting

| Error | Cause / fix |
|---|---|
| `400 MISSING_PARAM: Owner ID cannot be empty` | The list GET requires `?ownerId={userId}`. |
| `405 Method Not Allowed` on `/tableau/dataAlerts/{id}` | Single-alert GET unsupported — list and filter by `id` client-side. |
| `401 Bad_OAuth_Token` | Token expired — `sf api request rest` refreshes automatically on the next call. |
| `404` on `/wave/dataAlerts` or `/analytics/dataAlerts` | Wrong namespace — AHM alerts live under `/tableau/dataAlerts`. |
| `FUNCTIONALITY_NOT_ENABLED: [Wave]` | Unrelated — `/tableau/dataAlerts` does not require Wave / CRM Analytics. |
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall