forcedotcom/sf-skills / plugins/builder/agentforce-adlc/skills/agentforce-observe
agentforce-observe
VerifiedAnalyze production Agentforce agent behavior using session traces and Data Cloud, and manage Agent Health Monitoring (AHM) alerts. TRIGGER when: user queries STDM session data or Data Cloud trace records; investigates production agent failures, regressions, or performance issues; asks about session traces, conversation logs, or agent metrics; wants to reproduce a reported production issue in preview; runs findSessions or trace analysis queries; creates, lists, updates, or deletes an AHM data alert on an agent metric (escalation rate, deflection, etc.); asks why an alert is not firing or wants to check whether alerts have fired (notification counts, or the per-alert Incidents view). DO NOT TRIGGER when: user creates, modifies, or debugs .agent files during development (use agentforce-generate); writes or runs test specs (use agentforce-test); uses sf agent preview for local development iteration; deploys or publishes agents.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 21.9 KB |
| assets/apex/AgentforceOptimizeService.cls | other | 61.9 KB |
| assets/apex/AgentforceOptimizeService.cls-meta.xml | config | 174 B |
| references/ahm-alerts.md | markdown | 52.1 KB |
| references/improve-reference.md | markdown | 16.9 KB |
| references/issue-classification.md | markdown | 14.7 KB |
| references/reproduce-reference.md | markdown | 5.9 KB |
| references/stdm-queries.md | markdown | 15.0 KB |
| references/stdm-schema.md | markdown | 10.7 KB |
references/ahm-alerts.md
View on GitHub# AHM Alerts Reference — Agent Health Monitoring Data Alerts
End-to-end procedures for creating, listing, deleting, and inspecting **Agent Health Monitoring (AHM)** data alerts, and for verifying the metric values behind them. This is the same functionality the AHM Setup UI performs at `/lightning/n/standard-AgentforceStudio?c__nav=alerts`, but driven entirely from the `sf` CLI.
Use this reference when the user wants to:
- Create / list / delete an AHM alert on an agent metric (escalation rate, deflection rate, etc.)
- Understand why an alert isn't firing
- Inspect triggered alert notifications (trigger history)
- Discover the alertable SDMs, metrics, and agent filter values on an org
---
## Contents
1. [CLI-first — how commands are expressed here](#cli-first--how-commands-are-expressed-here)
2. [The endpoint: `tableau/dataAlerts`](#the-endpoint-tableaudataalerts)
3. [List and describe alerts](#list-and-describe-alerts)
4. [Prerequisite: confirm the three alertable SDMs](#prerequisite-for-creating-alerts-confirm-the-three-alertable-sdms)
5. [Enumerate `_mtc` metrics and screen for time-grain fit](#enumerate-_mtc-metrics-and-screen-for-time-grain-fit)
6. [List the agents available for alert filters](#list-the-agents-available-for-alert-filters)
7. [Create an alert](#create-an-alert) — POST vs GET field names, `filterContext`, thresholds, full schema, enums, troubleshooting
8. [Update an alert](#update-an-alert) — PUT in place
9. [Delete an alert](#delete-an-alert)
10. [Trigger history — inspect triggered notifications](#trigger-history--inspect-triggered-notifications)
11. [Verify metric value via Semantic Engine Gateway](#verify-metric-value-via-semantic-engine-gateway-condensed)
12. [Global troubleshooting](#global-troubleshooting)
---
## CLI-first — how commands are expressed here
Every operation is expressed through **`sf api request rest`**, which authenticates and executes with the org credentials the CLI already holds (`-o <org>`).
```bash
# Generic shape used throughout this reference
sf api request rest "<path>" -o <org> # GET
sf api request rest "<path>" -X POST -H "Content-Type: application/json" -b "@$body" -o <org>
sf api request rest "<path>/<id>" -X DELETE -o <org>
```
`sf api request rest` supports `-X/--method` (GET|POST|PUT|PATCH|DELETE|…), `-H/--header`, and `-b/--body` (`@file` to read a file, `-` for stdin, `""` for empty). Pipe the output to `python3 -m json.tool` to pretty-print.
> **Request bodies go in a private temp file — never a fixed `/tmp/<name>.json`.** The POST bodies below carry org data (SDM/agent names, user IDs, alert config). A predictable, world-readable path in the shared `/tmp` namespace invites info disclosure, symlink clobbering, and a TOCTOU swap between the `cat >` write and the `-b @file` read. Create a per-invocation `0600` file, pass it, and remove it on exit:
>
> ```bash
> body=$(mktemp) && chmod 600 "$body"
> trap 'rm -f "$body"' EXIT
> # cat > "$body" <<'JSON' … then: -b "@$body"
> ```
>
> Every heredoc example below assumes this `body=$(mktemp)` idiom (reuse the same `trap` for the run) and writes to `"$body"` rather than a literal `/tmp/*.json`.
> **Notes.**
> - **The org must already be authenticated — these commands do not launch a login.** Every call uses the credentials the CLI already holds for `-o <org>`. Pre-flight with `sf org display --target-org <org>` (or `sf org list`); if the alias is missing/unauthenticated, `sf` fails with `No authorization information found for <alias>` rather than prompting. Authenticate with `sf org login web --alias <alias>` — a browser flow the **user** runs (an agent can't complete it; in Claude Code prefix the command with `!`). An **expired** token auto-refreshes on the next call; a **missing/removed** auth does not.
> - There is **no dedicated `sf agent alert …` subcommand today** — the generic `sf api request rest` wrapper is the CLI surface for AHM alerts. If/when dedicated subcommands ship, prefer them and their `--help`.
> - `sf api request rest` is GA and bundled with the Salesforce CLI (no extra plugin required); it is the sole API surface used throughout this reference.
> - **API version.** Every endpoint below uses **`/services/data/v66.0/`**. For these resources the data API version is largely orthogonal to resource availability — v66.0 is the standardized floor here and `v68.0` also works on current orgs. Only raise the version if you need a field added in a later release; if any call unexpectedly returns `404`, retry at the org's current API version before assuming the resource is missing.
> - Reuse **Phase 0 (Discover Data Space)** from the main skill for the `dataspace` value used by the Data Cloud SQL and semantic queries below (default: `default`).
### Resolve the owner user ID
The list endpoint requires an `ownerId`. Resolve the current user (or another user by username):
```bash
# Current user (preferred — no user-supplied input)
USER_ID=$(sf org display user --target-org <org> --json \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["id"])')
# Another user, supplied out-of-band as a validated Salesforce User Id (15 or 18 chars).
# Prefer an Id over a username — it needs no query and can't inject.
USER_ID="005XXXXXXXXXXXXXXX"
[[ "$USER_ID" =~ ^[a-zA-Z0-9]{15,18}$ ]] || { echo "Not a valid User Id" >&2; exit 1; }
```
**Do not** build the owner query by interpolating a free-text username into `--query "… Username = '<value>'"`. That value crosses two hostile boundaries at once — the SOQL string literal (`x' OR IsActive=true LIMIT 1` subverts the predicate → resolves an arbitrary user's Id) and the surrounding double-quoted shell word (`$(…)`/backticks execute before `sf` runs). `agentforce-observe` reads production session data, so a username can be attacker-influenced, not just operator-typed.
If a username lookup is genuinely unavoidable, first reject anything with quoting/shell metacharacters, then build the SOQL in a `python3` snippet that reads the value from an environment variable and escapes the literal — never inline in a double-quoted `--query`:
```bash
UNAME='other@example.com'
# Reject any quote / backtick / $ / backslash / ; before the value is used.
printf '%s' "$UNAME" | LC_ALL=C grep -q "[\"'\`\$\\;]" && { echo "Reject: unsafe chars in username" >&2; exit 1; }
# UNAME is already validated above (no quotes/backslashes/shell metachars), so interpolate it directly.
SOQL=$(UNAME="$UNAME" python3 -c \
'import os;print("SELECT Id FROM User WHERE Username = \x27%s\x27" % os.environ["UNAME"])')
USER_ID=$(sf data query --target-org <org> --json --query "$SOQL" \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["records"][0]["Id"])')
```
---
## The endpoint: `tableau/dataAlerts`
AHM alerts are a special flavor of Salesforce data alerts — `dataAlertType: "agenthealthmonitoring"` — exposed under a dedicated REST resource:
```text
/services/data/v66.0/tableau/dataAlerts
```
They are **not** served from `/wave/dataAlerts` (CRM Analytics) or `/analytics/dataAlerts`. The "tableau" namespace here is Tableau Next / Data Cloud analytics, distinct from classic Wave — `/tableau/dataAlerts` does **not** require Wave / CRM Analytics.
| Method | Path | Purpose |
|--------|------|---------------------------------------------|
| GET | `/services/data/v66.0/tableau/dataAlerts?ownerId={userId}` | List alerts for a user (requires `ownerId`) |
| POST | `/services/data/v66.0/tableau/dataAlerts` | Create |
| PUT | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Update (same input representation as POST) |
| DELETE | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Delete |
| GET | `/services/data/v66.0/tableau/dataAlerts/{alertId}` | Not supported — 405 Method Not Allowed |
Single-alert GET is not supported — list all and filter client-side. Omitting `ownerId` on the list GET returns `400 MISSING_PARAM: Owner ID cannot be empty`. There is no "list all alerts across the org" shape; always scope to a user.
---
## List and describe alerts
```bash
sf api request rest "/services/data/v66.0/tableau/dataAlerts?ownerId=$USER_ID" -o <org> \
| python3 -m json.tool
```
Example response (`totalSize: 1`):
```json
{
"dataAlerts": [
{
"id": "3VRxx0000001xxxxAY",
"alertName": "te1_AHM_High_AHM_Escalation_Rate_mtc::Service_Agent_Analytics_SDM_1f8_AHM_Above_0.1_AHM_All",
"dataAlertType": "agenthealthmonitoring",
"createdDate": "2026-05-01T23:01:02.000Z",
"schedule": {"type": "everynminutes", "minuteLevelFrequency": 1},
"thresholds": {
"conditions": [{
"leftOperand": {
"type": "insights",
"factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
"filterContext": [],
"insightType": "popc",
"metricId": "1HUxx000000xxxx4AE",
"modelApiNameOrId": "2SMxx000000xxxx4AY",
"timeContext": {"operator": "LastNHours", "values": ["1"]}
},
"operator": "greaterorequal",
"rightOperand": {"type": "rawvalue", "dataType": "number", "value": "0.1"}
}],
"customLogicalOperation": "1"
},
"deliveryConfigurations": {
"receivers": [
{"type": "notification", "recipients": ["005xx000001xxxxAAC"]},
{"type": "email", "recipients": ["005xx000001xxxxAAC"]}
]
}
}
],
"totalSize": 1
}
```
Observations from a live response:
1. **`alertName` is an encoded form** — `_AHM_`-separated: `<freeText>_AHM_<severity>_AHM_<metricApiName>::<sdmApiName>_AHM_<operator>_<threshold>_AHM_<scope>`.
2. **`modelApiNameOrId` can be an ID** (e.g. `2SMxx…`), not always an API name.
3. **`filterContext: []`** even when the alert name indicates filters — this is a **known bug**. The real filters live in the **sub-metric** referenced by `metricId`.
4. **`schedule.minuteLevelFrequency: 1`** — fires every minute (aggressive; useful for test/demo).
### Retrieve the sub-metric to see the actual filters
```bash
METRIC_ID="1HUxx000000xxxx4AE"
MODEL="Service_Agent_Analytics_SDM" # or the ID from modelApiNameOrId
sf api request rest \
"/services/data/v66.0/ssot/semantic/models/${MODEL}/sub-metrics/${METRIC_ID}" -o <org> \
| python3 -m json.tool
```
HTTP 200 → it's a filtered sub-metric; `filters[]` holds the real agent-name / agent-type conditions.
---
## Prerequisite for creating alerts: confirm the three alertable SDMs
> **Scope rule.** AHM data alerts target **exactly three** semantic data models. **Ignore every other SDM** returned by the endpoint — including `Agent_Health_Monitoring_SDM`, dataset-specific models, or custom models. If the metric you want isn't in one of the three below, the answer is "not supported," not "try another SDM."
1. **Agentforce Analytics Foundations** — base model. API name `sfm_Agentforce_Analytics_Foundations` (stable). Exposes only `_clc`, no `_mtc`.
2. **Employee Agent Analytics SDM** — match by **label** `Employee Agent Analytics SDM` or **`app` = `Employee_Agent_Analytics`**.
3. **Service Agent Analytics SDM** — match by **label** `Service Agent Analytics SDM` or **`app` = `Service_Agent_Analytics`**.
> **Never hard-code the `_1f8` (or any other) `apiName` suffix.** It's a content-hash / provisioning token that changes when the app template is re-provisioned or moved to another org. Always discover the live `apiName` by listing models and matching on the stable fields (`label`, `app`), then read `apiName` back out.
### List all semantic models
```bash
sf api request rest "/services/data/v66.0/ssot/semantic/models" -o <org> \
| python3 -m json.tool
```
Example `items[]` entry:
```json
{
"apiName": "Service_Agent_Analytics_SDM_1f8",
"app": "Service_Agent_Analytics",
"label": "Service Agent Analytics SDM",
"id": "2SMxx000000xxxx4AY",
"dataspace": "default",
"baseModels": [
{"apiName": "sfm_Agentforce_Analytics_Foundations", "label": "Agentforce Analytics Foundations"}
]
}
```
Key fields: `label` (stable match), `apiName` (goes into `modelApiNameOrId`, has the suffix), `app`/`sourceCreationName` (stable app identifiers), `id` (SDM record ID — needed for the semantic-engine gateway).
### Provisioning check — the three only, ignore everything else
```bash
export ORG=<org>
sf api request rest "/services/data/v66.0/ssot/semantic/models" -o "$ORG" | python3 <<'PY'
import json, sys
models = json.load(sys.stdin).get("items", [])
# Allowlist: exactly the three AHM-alertable SDMs. Match by stable fields only.
IN_SCOPE = [
("Agentforce Analytics Foundations", None),
("Employee Agent Analytics SDM", "Employee_Agent_Analytics"),
("Service Agent Analytics SDM", "Service_Agent_Analytics"),
]
def match(m, label, app):
return m.get("label") == label or (app and m.get("app") == app)
print(f"{'SDM (in scope)':40} {'apiName':45} {'id':25} status")
print("-" * 120)
resolved_ids = set()
for label, app in IN_SCOPE:
hits = [m for m in models if match(m, label, app)]
if not hits:
print(f"{label:40} {'(not found)':45} {'':25} Not provisioned — cannot alert on this SDM")
continue
m = hits[0]
resolved_ids.add(m.get("id"))
print(f"{label:40} {m.get('apiName','?'):45} {str(m.get('id','?')):25} provisioned")
ignored = [m for m in models if m.get("id") not in resolved_ids]
print(f"\nIgnored — out of scope ({len(ignored)} model(s)). Do NOT alert on these:")
for m in ignored:
print(f" • {m.get('label','?')} (apiName={m.get('apiName','?')})")
if not ignored:
print(" (none)")
PY
```
Interpreting the output:
- All three provisioned → proceed to create alerts; use the `apiName` values as-is for `modelApiNameOrId`.
- Any Not provisioned → that SDM's app template is not provisioned on this org. Alerts cannot be created against it until the template is deployed (a provisioning task, not a CLI workaround).
- Anything "Ignored" → acknowledge and move on. **Do not** substitute an ignored SDM for a missing in-scope one.
Troubleshooting:
- **Only Foundations shows up** — the Employee/Service app templates haven't been provisioned; they must be deployed via the Analytics app template flow first.
- **`FUNCTIONALITY_NOT_ENABLED` on `/ssot/semantic/models`** — Data Cloud / SSOT is not provisioned; none of the alertable SDMs can exist.
- **API name doesn't match `modelApiNameOrId` in an existing alert** — the alert was created against an older provisioning hash. Delete and recreate against the current `apiName`.
---
## Enumerate `_mtc` metrics and screen for time-grain fit
The alertable atoms inside each SDM are **semantic metrics** whose API names end in `_mtc`. These differ from the `_clc` calculated measurements:
| Endpoint | Object | Suffix | What it is |
|---|---|---|---|
| `…/calculated-measurements` | Calculated measurement | `_clc` | Low-level aggregation at the SDM layer (raw ingredient). |
| `…/metrics` | Semantic metric | `_mtc` | Wraps a `_clc` with time dimension, grains, insight settings. **This is what data alerts reference.** |
| `…/sub-metrics` | Sub-metric | id `1HU…` | A filtered instantiation of a `_mtc`. Most real alerts' `metricId` is a sub-metric, not the bare `_mtc`. |
### List all `_mtc` across the three SDMs
```bash
export ORG=<org>
python3 <<'PY'
import json, subprocess
def rest(path):
out = subprocess.check_output(["sf","api","request","rest",path,"-o",__import__("os").environ["ORG"]])
return json.loads(out)
models = rest("/services/data/v66.0/ssot/semantic/models").get("items", [])
IN_SCOPE_APPS = {
"Agentforce Analytics Foundations": None,
"Employee Agent Analytics SDM": "Employee_Agent_Analytics",
"Service Agent Analytics SDM": "Service_Agent_Analytics",
}
resolved = {}
for m in models:
for label, app in IN_SCOPE_APPS.items():
if m.get("label") == label or (app and m.get("app") == app):
resolved[label] = m["apiName"]
break
for label, api in resolved.items():
metrics = rest(f"/services/data/v66.0/ssot/semantic/models/{api}/metrics").get("metrics", [])
mtc = [m for m in metrics if m.get("apiName","").endswith("_mtc")]
print(f"\n=== {label} ({api}) — {len(mtc)} _mtc metric(s) ===")
for m in mtc:
grains = ",".join(m.get("timeGrains", [])) or "(no timeGrains)"
print(f" {m['apiName']:40} | {m.get('label',''):30} | id={m.get('id','')} | grains=[{grains}]")
if not mtc:
print(" (none — e.g. Foundations exposes only _clc)")
PY
```
On a typical org, Foundations exposes **0** `_mtc`; Service Analytics SDM exposes ~14 (`Escalation_Rate_mtc`, `Deflection_Rate_mtc`, `Abandonment_Rate_mtc`, `Engagement_Rate_mtc`, `Agent_Health_Score_mtc`, `Total_Sessions_mtc`, …); Employee Analytics SDM exposes ~9 (`Stickiness_Rate_mtc`, `Total_Unique_Users_mtc`, `Weekly_Active_Users_mtc`, …). All declare `timeGrains = [Day, Week, Month, Quarter, Year]` — none declare Hour/Minute, but the `dataAlerts` API doesn't enforce this (the alert's `timeContext.operator` is independent).
Pull the `id` of the chosen `_mtc` — it goes into `thresholds.conditions[].leftOperand.metricId` when creating an alert.
### Screening: does the metric make sense at a short window?
The SDM doesn't block nonsense; apply judgement:
| Metric shape | Smallest useful window | Short-window (≤ 1h) alert? |
|---|---|---|
| **Ratios / rates** — Engagement, Deflection, Abandonment, Escalation, Stickiness | 15 min – 1 h | Yes — self-normalizing. Add a minimum-volume guard. |
| **Averages / times** — Avg Time to Deflection/Escalation/Execution, Avg Quality Score | ≥ 1 h | Borderline at 1–4h; daily is fine. Not at 15 min. |
| **Cumulative totals** — Total/Deflected/Escalated/Abandoned/Engaged Sessions, Total Actions | ≥ 1 h, typically ≥ 1 day | No — noisy at short windows — alert on the corresponding **rate** instead. |
| **Cardinality counts** — Unique Users, Weekly Active Users, Total Employee Agents | ≥ 1 day | No — smallest sensible window Day (DAU) / Week (WAU). |
| **Composite scores** — Agent Health Score | default ≥ 1 day | Depends on inputs; default no. |
> **Minimum-volume guard.** Even for ratios, check the denominator: a 100% error rate on 1 session in 15 min is noise. The API has no built-in guard — pre-filter via a sub-metric requiring N > threshold sessions, or combine a rate threshold AND an absolute-count threshold in `thresholds.conditions[]`.
Use short-window rate metrics with `schedule.minuteLevelFrequency ≤ 60` and `timeContext.operator = LastNHours|LastNMinutes`; use daily+ metrics only with `minuteLevelFrequency ≥ 1440` and `LastNDays`/`LastNWeeks`.
---
## List the agents available for alert filters
The UI Create-Alert modal scopes an alert to an **Agent Name** or **Agent Type**. Those values come from **live session-tracing data (Data Cloud DLO)**, not from agent metadata.
> **For AHM alerts, use the DLO query (ground truth) below.** If a name appears in `BotDefinition` but has never run a session, the filter selects zero sessions and the alert never fires. A name in the DLO without a `BotDefinition` (demo/external agents) is still alertable — the filter matches strings, not IDs.
>
> **Cross-link:** the DLO object below, `ssot__AiAgentSessionParticipant__dlm`, is the same DMO documented in [`stdm-schema.md`](stdm-schema.md). Note the AHM filter uses `ssot__AiAgentApiName__c` here, which is **not** the same as the `GenAiPlannerDefinition` `MasterLabel`/`DeveloperName` the main skill resolves for STDM `findSessions` — grab the filter value from this query, don't reconstruct it.
### DLO query (ground truth)
Write the SQL body to a private temp file (see the `mktemp` idiom above) to avoid shell-escaping, then POST it:
```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
"sql": "SELECT ssot__AiAgentApiName__c AS agent_name, ssot__AiAgentType__c AS agent_type, COUNT(*) AS participant_rows FROM ssot__AiAgentSessionParticipant__dlm WHERE ssot__AiAgentSessionParticipantRole__c != 'USER' GROUP BY ssot__AiAgentApiName__c, ssot__AiAgentType__c ORDER BY agent_name",
"rowLimit": 500,
"adaptiveTimeout": 1
}
JSON
sf api request rest "/services/data/v66.0/ssot/query-sql?dataspace=default" \
-X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
| python3 -m json.tool
```
The `ssot__AiAgentSessionParticipantRole__c != 'USER'` clause filters out the human side of the conversation. `GROUP BY` already yields one row per distinct `(agent_name, agent_type)` pair — no `DISTINCT` needed. In the `data` array: column 0 = `agent_name` (→ `AI_Agent_Api_Name` filter values), column 1 = `agent_type`, column 2 = `participant_rows` (volume sanity check — an agent with a handful of rows is too sparse for a 15-min alert).
### Metadata cross-reference (fallback)
```bash
sf data query --target-org <org> \
--query "SELECT Id, DeveloperName, MasterLabel, AgentType FROM BotDefinition ORDER BY DeveloperName"
```
Not required for alerting. `AiAgent` sObject is often not queryable via REST — don't rely on it.
### Next-Gen Authoring Bundles API (what agents are defined)
Works even where `BotDefinition`/`AiAgent` sObjects are not queryable (common on demo orgs):
```bash
sf api request rest "/services/data/v66.0/nextgen-authoring/bundles" -o <org> | python3 -m json.tool
# names only:
sf api request rest "/services/data/v66.0/nextgen-authoring/bundles" -o <org> \
| python3 -c 'import sys,json; [print(b.get("apiName","?")) for b in json.load(sys.stdin).get("bundles", [])]'
```
`apiName` is the agent's developer name (the value usable as an alert filter); `isLegacy: false` = new Next-Gen agent. Requires `NextGenAuthoring.orgHasNextGenAgentAuthoringEnabled` + `…userCanAccessNextGenAgentAuthoring`, else 403 (the `.get("bundles", [])` above then prints nothing rather than raising).
**Use the DLO query (ground truth) for filter values.** The metadata cross-reference and Bundles API are for discovery/cross-ref only. An agent listed there but absent from the DLO has never run a session — alerting on it matches zero rows until sessions arrive.
Troubleshooting:
- **`query-sql` 400 / FUNCTIONALITY_NOT_ENABLED** — SSOT/Data Cloud not enabled; no agent-tracing data exists and alerts can't fire.
- **Zero rows** — no sessions have landed. Confirm session-tracing is on and data has flowed.
- **UI dropdown shows values not in the DLO query** — check the `dataspace`; change `?dataspace=` if the alert is scoped to a non-`default` space.
---
## Create an alert
`POST /services/data/v66.0/tableau/dataAlerts`. Inputs come from the sections above: `modelApiNameOrId` (SDM `apiName`/`id`), `metricId` (`_mtc` `id`), and `filterContext` (agent name/type from the DLO query above).
### POST field names ≠ GET field names
The POST input representation uses **different field names and casing** from the GET response:
| POST (input) | GET (output) | Notes |
|---|---|---|
| `utterance` | `alertName` | POST field is `utterance`; response returns `alertName` |
| `type: "Metric"` (leftOperand) | `type: "insights"` | PascalCase in, lowercase out |
| `type: "RawValue"` (rightOperand) | `type: "rawvalue"` | same pattern |
| `type: "EveryNMinutes"` (schedule) | `type: "everynminutes"` | same |
| `type: "Notification"` / `"Email"` | `type: "notification"` / `"email"` | same |
| `operator: "GreaterOrEqual"` | `operator: "greaterorequal"` | same |
| `insightType: "Popc"` | `insightType: "popc"` | same |
**Rule of thumb:** all `type` discriminators and enum values in the POST body use **PascalCase**; the GET response lowercases everything. Copying a GET response and POSTing it back fails with `JSON_PARSER_ERROR`.
### filterContext field-name format
Filter field names use **dot notation**: `{tableApiName}.{fieldApiName}`. The metric's `additionalDimensions` lists the available dimensions as `tableFieldReference` objects — concatenate `tableApiName` + `.` + `fieldApiName`:
```text
tableApiName: "Agent_API_Name_lv"
fieldApiName: "AI_Agent_Session_Participant2_AI_Agent_Api_Name"
→ filterContext fieldName: "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name"
```
Using just the `fieldApiName` fails with `INTERNAL_ERROR: Validation Failed: Invalid calculated Field`.
### Sub-metric auto-creation
When you POST with a non-empty `filterContext`, the backend **auto-creates a sub-metric** (ID prefix `1HU…`) from the bare `_mtc` id plus your filters. The response's `metricId` is the new sub-metric id, not the `_mtc` id you sent. You don't create sub-metrics manually. (The response's `filterContext` comes back `[]` — the known bug; fetch the sub-metric to see the real filters.)
### Threshold values are raw ratios, not display percentages
Rate metrics are a **0–1 ratio** (the UI multiplies by 100 for display; the alert threshold operates on the raw value):
| You want to alert at | Threshold `value` |
|---|---|
| 1% | `"0.01"` |
| 5% | `"0.05"` |
| 50% | `"0.5"` |
| 100% | `"1"` |
> `value: "1"` means ≥ 100%, not ≥ 1%. An alert with `"1"` on a rate metric is either a test alert designed to always fire, or a misconfiguration.
### Full POST schema
```json
{
"utterance": "<alert name — encoded or freeform string>",
"dataAlertType": "agenthealthmonitoring",
"schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": <integer minutes> },
"content": { "type": "Metric", "modelApiNameOrId": ["<SDM ID or apiName>"] },
"thresholds": {
"conditions": [
{
"leftOperand": {
"type": "Metric",
"modelApiNameOrId": "<SDM ID or apiName>",
"metricId": "<_mtc id>",
"insightType": "Popc",
"factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
"params": {},
"filterContext": [
{ "fieldName": "<table.field dot notation>", "operator": "Equals", "values": ["<filter value>"] }
],
"timeContext": { "operator": "LastNHours", "values": ["<hours>"] }
},
"operator": "GreaterOrEqual",
"rightOperand": { "type": "RawValue", "dataType": "Number", "value": "<threshold as string>" }
}
],
"customLogicalOperation": "1"
},
"deliveryConfigurations": {
"receivers": [
{ "type": "Notification", "recipients": ["<userId>"] },
{ "type": "Email", "recipients": ["<userId>"] }
]
}
}
```
### Example: Escalation Rate ≥ 1% for agent `te1`, checked every 15 minutes
Build the body in a private temp file (see the `mktemp` idiom above), then POST it. The literal-value heredoc below shows the exact shape (verified against a real org) with **constant** example values — safe to paste as-is because nothing is substituted:
```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
"utterance": "te1_AHM_High_AHM_Escalation_Rate_mtc::Service_Agent_Analytics_SDM_1f8_AHM_Above_0.01_AHM_te1",
"dataAlertType": "agenthealthmonitoring",
"schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": 15 },
"content": { "type": "Metric", "modelApiNameOrId": ["2SMSG000000a6TV4AY"] },
"thresholds": {
"conditions": [
{
"leftOperand": {
"type": "Metric",
"modelApiNameOrId": "2SMSG000000a6TV4AY",
"metricId": "1DOSG0000051KAm4AM",
"insightType": "Popc",
"factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
"params": {},
"filterContext": [
{ "fieldName": "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name", "operator": "Equals", "values": ["te1"] }
],
"timeContext": { "operator": "LastNHours", "values": ["1"] }
},
"operator": "GreaterOrEqual",
"rightOperand": { "type": "RawValue", "dataType": "Number", "value": "0.01" }
}
],
"customLogicalOperation": "1"
},
"deliveryConfigurations": {
"receivers": [
{ "type": "Notification", "recipients": ["<USER_ID>"] },
{ "type": "Email", "recipients": ["<USER_ID>"] }
]
}
}
JSON
sf api request rest "/services/data/v66.0/tableau/dataAlerts" \
-X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
| python3 -m json.tool
```
**When the agent name, `utterance`, filter value, threshold, or recipient come from data — not typed by hand — do not string-substitute them into the heredoc.** An agent name sourced from live DLO/session data can contain a `"` that breaks out of the JSON string and injects sibling fields (e.g. an attacker-controlled `Email` receiver that exfiltrates notifications, or a corrupted threshold that silently disables the alert). Reject values containing `"`/`\`, then let a JSON-aware builder encode everything — the body is byte-safe regardless of what the values contain:
```bash
# AGENT_NAME / UTTERANCE / THRESHOLD (raw ratio) / MODEL_ID / METRIC_ID / USER_ID come from earlier discovery.
case "$AGENT_NAME$UTTERANCE" in *[\"\\]*) echo "Reject: quote/backslash in name or utterance" >&2; exit 1;; esac
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
AGENT_NAME="$AGENT_NAME" UTTERANCE="$UTTERANCE" THRESHOLD="$THRESHOLD" \
MODEL_ID="$MODEL_ID" METRIC_ID="$METRIC_ID" USER_ID="$USER_ID" python3 - > "$body" <<'PY'
import os, sys, json
e = os.environ
json.dump({
"utterance": e["UTTERANCE"],
"dataAlertType": "agenthealthmonitoring",
"schedule": {"type": "EveryNMinutes", "minuteLevelFrequency": 15},
"content": {"type": "Metric", "modelApiNameOrId": [e["MODEL_ID"]]},
"thresholds": {"conditions": [{
"leftOperand": {"type": "Metric", "modelApiNameOrId": e["MODEL_ID"], "metricId": e["METRIC_ID"],
"insightType": "Popc", "factKey": "FACT_KEY_TARGET_PERIOD_VALUE", "params": {},
"filterContext": [{"fieldName": "Agent_API_Name_lv.AI_Agent_Session_Participant2_AI_Agent_Api_Name",
"operator": "Equals", "values": [e["AGENT_NAME"]]}],
"timeContext": {"operator": "LastNHours", "values": ["1"]}},
"operator": "GreaterOrEqual",
"rightOperand": {"type": "RawValue", "dataType": "Number", "value": e["THRESHOLD"]}}],
"customLogicalOperation": "1"},
"deliveryConfigurations": {"receivers": [
{"type": "Notification", "recipients": [e["USER_ID"]]},
{"type": "Email", "recipients": [e["USER_ID"]]}]},
}, sys.stdout)
PY
sf api request rest "/services/data/v66.0/tableau/dataAlerts" \
-X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
| python3 -m json.tool
```
The response (HTTP 200) returns `id` (the alert id, prefix `3VR…`), and a `metricId` with prefix `1HU…` — the auto-created sub-metric, not the `1DO…` bare metric you sent. Response `filterContext` is `[]` (known bug); confirm the real filter via the sub-metric fetch shown earlier. All enum values in the response are lowercase.
### Example: unfiltered alert (all agents)
Same body, but `"filterContext": []` and (typically) `minuteLevelFrequency: 60` — no sub-metric is created; the alert references the bare `_mtc` id directly.
### Available enum values
- **Schedule types:** `"EveryNMinutes"` (+`minuteLevelFrequency`), `"Daily"` (+`hoursOfDay`), `"Weekly"` (+`hoursOfDay`,`daysOfWeek`).
- **Condition operators:** `"Equals"`, `"NotEqual"`, `"GreaterThan"`, `"GreaterOrEqual"`, `"LessThan"`, `"LessOrEqual"`.
- **Value types:** `"Metric"` (leftOperand); `"RawValue"` (rightOperand) with `dataType` `"Number"`/`"Text"`/`"Percent"`.
- **Receiver types:** `"Notification"` / `"Email"` (recipients = user IDs); `"Slack"` (recipients = Slack channel IDs).
- **Insight types:** `"Popc"` (period-over-period — used by AHM), `"RiskyMonopoly"`, `"Unspecified"`.
- **Filter operators:** `"Equals"`, `"DoesNotEquals"`, `"LessThan"`, `"GreaterThan"`, `"LessOrEqual"`, `"GreaterOrEqual"`, `"Between"`, `"IsNull"`, `"IsNotNull"`, `"Contains"`, `"DoesNotContain"`, `"StartsWith"`, `"EndsWith"`, `"In"`, `"NotIn"`.
- **Time-context operators:** `"LastNHours"`, `"LastNDays"`, `"LastNWeeks"`, `"LastNMonths"`, `"LastNQuarters"`, `"LastNYears"`, `"LastNMinutes"`.
### Constructing the `utterance` (alert name)
Conventional encoded format the AHM UI parses (the backend accepts arbitrary strings):
```text
{freeText}_AHM_{severity}_AHM_{metricApiName}::{sdmApiName}_AHM_{operator}_{threshold}_AHM_{scope}
```
`severity` = `High`/`Medium`/`Low`; `operator` = `Above`/`Below`; `scope` = `All` (no filter) or the filter value. Using this format lets the AHM UI display the alert correctly.
### Troubleshooting
- **`JSON_PARSER_ERROR: Unrecognized field`** — using a GET-response field name in the POST. Common: `alertName` → use `utterance`; `intervalInMinutes` → `minuteLevelFrequency`.
- **`JSON_PARSER_ERROR: Could not resolve type id`** — wrong casing on a `type` discriminator. Use PascalCase (`"Metric"`, `"RawValue"`, `"EveryNMinutes"`, …).
- **`INTERNAL_ERROR: Validation Failed: Invalid calculated Field`** — filter `fieldName` missing the table prefix; use dot notation.
- **`400` with no clear message** — `content.modelApiNameOrId` must be an **array** (`["2SM…"]`), not a string.
- **Alert created but never fires** — the agent name in `filterContext.values` must exactly match a DLO value (from the DLO query in "List the agents available for alert filters"). If the agent has never had a session, there's no data to alert on.
---
## Update an alert
`PUT /services/data/v66.0/tableau/dataAlerts/{alertId}` updates an alert **in place** — same `id`, no delete + recreate. It takes the **same input representation as POST** (PascalCase `type` discriminators, `utterance` not `alertName`, `RawValue`/`Metric`/`EveryNMinutes`, threshold as a raw-ratio string, etc. — see [Create an alert](#create-an-alert)). Send the **full** alert body, not a partial patch: the representation is replaced, so include every field you want to keep (schedule, content, all `thresholds.conditions`, `deliveryConfigurations`). Returns **HTTP 200** with the updated alert in GET-shape (lowercased enums); `id` is unchanged, `lastModifiedDate` is bumped.
> **Addressed by alert id.** Like `DELETE`, `PUT` targets the alert via its `{alertId}` in the **path** (`/tableau/dataAlerts/{alertId}`) — the id is *not* in the body. A metricId (`1HU…`) in the path 404s; use the alert id (`3VR…`).
>
> **Read-modify-write.** Fetch the current alert first (`GET tableau/dataAlerts?ownerId=$USER_ID`, filter by `id`), change only the fields you intend to, and PUT the whole thing back — translating the GET response's field names/casing to the POST input names (`alertName`→`utterance`, lowercase `type`s→PascalCase). Keep the existing `metricId` in the body: PUT preserves it (no sub-metric churn) as long as `filterContext` is unchanged.
### Example: change a threshold (e.g. `80` → `85`)
Build the full body in a private temp file (see the `mktemp` idiom above), then PUT it. This mirrors the POST schema with the alert's current values, changing only `rightOperand.value` (and, optionally, `utterance`):
```bash
ALERT_ID="3VRSG0000001uBR4AY"
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
"utterance": "Task Resolution Rate Less Than 85% for AgentRuntimeInsights262",
"dataAlertType": "agenthealthmonitoring",
"schedule": { "type": "EveryNMinutes", "minuteLevelFrequency": 10 },
"content": { "type": "Metric", "modelApiNameOrId": ["2SMSG000000bPXf4AM"] },
"thresholds": {
"conditions": [
{
"leftOperand": {
"type": "Metric",
"modelApiNameOrId": "2SMSG000000bPXf4AM",
"metricId": "1HUSG0000010Un74AE",
"insightType": "Popc",
"factKey": "FACT_KEY_TARGET_PERIOD_VALUE",
"params": {},
"filterContext": [],
"timeContext": { "operator": "LastNMinutes", "values": ["15"] }
},
"operator": "LessThan",
"rightOperand": { "type": "RawValue", "dataType": "Number", "value": "85" }
}
],
"customLogicalOperation": "1"
},
"deliveryConfigurations": {
"receivers": [
{ "type": "Notification", "recipients": ["005SG00000ZEQePYAX"] },
{ "type": "Email", "recipients": ["005SG00000ZEQePYAX"] }
]
}
}
JSON
sf api request rest "/services/data/v66.0/tableau/dataAlerts/$ALERT_ID" \
-X PUT -H "Content-Type: application/json" -b "@$body" -o <org> \
| python3 -m json.tool
```
When the new name/threshold/recipients come from **data** rather than being typed by hand, do not string-substitute into the heredoc — reject values containing `"`/`\` then build the JSON with the same `python3` JSON-dump idiom shown under [Create an alert](#create-an-alert) (the injection risk is identical for PUT).
**Verified (org `ahm-test-notifications`, 2026-09-02):** PUT on alert `3VRSG0000001uBR4AY` with the POST-style body above returned HTTP 200, changed `rightOperand.value` `80`→`85` and `alertName`, kept the same `id` and `metricId` (`1HUSG0000010Un74AE`), and bumped `lastModifiedDate` — a true in-place update. Reverting to `80` via a second PUT worked identically.
### Troubleshooting
- **Same `JSON_PARSER_ERROR` / casing / array-shape errors as POST** — the input grammar is identical; see [Create an alert → Troubleshooting](#troubleshooting).
- **`404` on the PUT** — confirm the `{alertId}` is a real alert `id` (`3VR…`) from the list; a metricId (`1HU…`) will 404. If the id is valid and it still 404s, retry at the org's current API version.
- **Fields silently reverting** — PUT replaces the representation; a field you omitted reverts to server default. Always send the full body (read-modify-write), not a partial patch.
---
## Delete an alert
```bash
ALERT_ID="3VRSG0000001kX34AI" # from the list or the create response
sf api request rest "/services/data/v66.0/tableau/dataAlerts/$ALERT_ID" -X DELETE -o <org> --include
```
Returns **HTTP 204** (No Content) on success — no response body. Use `--include` to see the status line.
---
## Trigger history — inspect triggered notifications
When an alert's condition is met it generates a **system notification** and optionally an **email** (per `deliveryConfigurations.receivers`). AHM alert notifications carry the notification `type` **`templatized_data_alert`** and their `target` field holds the **sub-metric id** (`1HU…`) — the same value as the alert's `thresholds.conditions[0].leftOperand.metricId`.
**Attribute by alert id only.** The notification's `targetPageRef` (an HTML-encoded JSON string) embeds the owning alert id under `state.c__alertId` (or `state.c__dataAlertId`, or nested `attributes.pageRef.state.c__alertId`/`c__dataAlertId`). That id is the **precise** attribution key — match on it. **Do not fall back to the `target == metricId` join**: it is only correct when there's one alert per metric — two alerts on the same bare `_mtc` (both with empty `filterContext`) share a `metricId`, so the metricId join returns the **union** of their firings and cannot tell them apart. A notification whose alert id can't be recovered is simply **not attributed** (see the null-handling note below), not routed through metricId.
### Extracting and matching the alert id
```python
import json, html
def extract_alert_id(n):
"""Return the owning alert id from a notification, or None. Degrades to None
(never raises) on missing / non-JSON / key-absent targetPageRef → 'no match'."""
ref = n.get("targetPageRef")
if not ref:
return None
try:
if isinstance(ref, str):
ref = json.loads(html.unescape(ref)) # API HTML-encodes the quotes (")
except Exception:
return None
top = (ref or {}).get("state") or {}
if top.get("c__dataAlertId"): return top["c__dataAlertId"]
if top.get("c__alertId"): return top["c__alertId"]
nested = (((ref or {}).get("attributes") or {}).get("pageRef") or {}).get("state") or {}
return nested.get("c__dataAlertId") or nested.get("c__alertId")
def ids_match(a, b):
"""15/18-char-safe Salesforce id equality."""
if not a or not b: # None from a failed extraction ⇒ no match, drop
return False
return a == b or a[:15] == b[:15] # 18-char = 15-char + 3-char checksum suffix
```
- **`None` extraction is a non-match, not an error.** A missing/malformed/key-absent `targetPageRef` yields `None`; `ids_match` returns `False` on a `None` operand, so the notification is simply **dropped** (not attributed to any alert) — nothing throws, and it is **not** routed through a metricId fallback.
- **Compare the first 15 chars.** The id embedded in `targetPageRef` is frequently the **15-char** form; the alert id from `GET tableau/dataAlerts` is the **18-char** form (15-char + 3-char case-insensitive checksum). Strict `===` fails on the same record (`3VRSG0000001vHB` vs `3VRSG0000001vHB4AY`); try exact first, then `a[:15] == b[:15]`.
- **Casing caveat.** The 15-char id is case-sensitive and the 18-char checksum encodes that casing; `ids_match` does **not** case-normalize, so truncating to 15 is correct only because both sides preserve the true casing (they do in this flow — neither source upper/lower-cases the id). Do not add a `.lower()` on either side.
- **Verified (org `ahm-test-notifications`):** all fired notifications carried `state.c__alertId` (18-char, e.g. `3VRSG0000001vHB4AY`) matching a listed alert 1:1; the nested branches and `c__dataAlertId` are defensive fallbacks for other page-ref shapes.
### The `X-UNS-Type-Filter: all` header is REQUIRED to see AHM notifications
The Connect Notifications API (`connect/notifications*`) **defaults to returning *custom* notification types only**. AHM/Tableau-Next alerts are **standard/platform** types (`templatized_data_alert`), so a plain call returns an empty list / empty types even when alerts have genuinely fired. **Always pass `-H "X-UNS-Type-Filter: all"`** on every `connect/notifications*` call to include standard/platform types:
```bash
# Status (counts). WITHOUT the header these are custom-types-only and read 0 for AHM.
sf api request rest "/services/data/v66.0/connect/notifications/status" \
-H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool
# List all notifications (includes AHM). The header is what surfaces them.
sf api request rest "/services/data/v66.0/connect/notifications" \
-H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool
# Which notification types the org exposes (verify templatized_data_alert is present).
sf api request rest "/services/data/v66.0/connect/notifications/types" \
-H "X-UNS-Type-Filter: all" -o <org> | python3 -m json.tool
```
> **Symptom of the missing header:** `connect/notifications` → `{ "notifications": [] }`, `connect/notifications/types` → `{ "notificationTypes": [] }`, and `status` → `unreadCount: 0` — even though the alerts fired and are visible in the UI bell. This empty result is a **false negative**, not proof of "not fired." (The UI bell uses the internal Aura controller `ui-notifications-components-notifications-controller.Notifications.getNotifications`, which reads the full store; the CLI must opt in with the header.)
Each notification object includes: `type`, `messageTitle` (metric + agent + current value), `messageBody` (`The current value (X) is above/below the alert condition (Y)`), `lastModified` (fire time), `read`/`seen`, `target` (the sub-metric id), and `targetPageRef` (an HTML-encoded JSON string). The **owning alert id** is in `targetPageRef.state.c__alertId` — that, not `target`, is the attribution key (see "Extracting and matching the alert id" above).
### Answering "get notifications" (default response)
When the user asks to get / list / check notifications for an org, report **both**:
1. **Status** — `unreadCount` / `unseenCount` from the `status` call (with the header).
2. **Notifications** — the list from the `notifications` call (with the header), summarized (title, body, fire time, type).
Filter to `type == "templatized_data_alert"` to isolate AHM/agent-health alerts (the type is shared with other templatized Tableau data alerts, so it is **not** AHM-exclusive — filter, don't assume).
### Answering "notifications for a specific alert"
There is **no per-alert notification endpoint** (`…/dataAlerts/{id}/notifications` → 404). Instead, fetch **all** notifications once, then **filter client-side by alert id** (exact — see "Extracting and matching the alert id" above):
1. Fetch all notifications with the header (above).
2. For each notification, `extract_alert_id(n)` and keep those where `ids_match(extracted, targetAlertId)` (15/18-char-safe). Those are the firings of that alert.
3. A notification whose `targetPageRef` is missing/unparseable (`extract_alert_id` → `None`) is **dropped**, not attributed. Do **not** fall back to a `target == metricId` join — it mis-attributes when two alerts share a metric.
Present title / current value / threshold / `lastModified` per firing.
```bash
# Attribute every notification to its owning alert (by alert id from targetPageRef — NOT metricId).
ORG=<org>
USER_ID=$(sf org display user --target-org "$ORG" --json | python3 -c 'import sys,json;print(json.load(sys.stdin)["result"]["id"])')
sf api request rest "/services/data/v66.0/tableau/dataAlerts?ownerId=$USER_ID" -o "$ORG" > /tmp/alerts.json
sf api request rest "/services/data/v66.0/connect/notifications" -H "X-UNS-Type-Filter: all" -o "$ORG" > /tmp/notifs.json
python3 - <<'PY'
import json, html
from collections import defaultdict
def extract_alert_id(n):
ref = n.get("targetPageRef")
if not ref:
return None
try:
if isinstance(ref, str):
ref = json.loads(html.unescape(ref))
except Exception:
return None
top = (ref or {}).get("state") or {}
if top.get("c__dataAlertId"): return top["c__dataAlertId"]
if top.get("c__alertId"): return top["c__alertId"]
nested = (((ref or {}).get("attributes") or {}).get("pageRef") or {}).get("state") or {}
return nested.get("c__dataAlertId") or nested.get("c__alertId")
def norm(x):
return x[:15] if x else x # 15/18-char-safe: truncate both sides to the 15-char id
alerts = json.load(open('/tmp/alerts.json')).get("dataAlerts", [])
notifs = json.load(open('/tmp/notifs.json')).get("notifications", [])
by_alert = {norm(a.get("id")): a for a in alerts} # key on the alert id (15-char-normalized)
groups = defaultdict(list)
for n in notifs:
groups[norm(extract_alert_id(n))].append(n) # None-key bucket = no recoverable alert id
for aid, a in by_alert.items():
ns = sorted(groups.get(aid, []), key=lambda x: x.get("lastModified", ""))
print(f"\nALERT: {a.get('alertName')} (id={a.get('id')}) -> {len(ns)} fired")
for n in ns:
print(" %s | %s" % (n.get("lastModified", "")[:19], html.unescape(n.get("messageTitle", ""))))
# notifications whose extracted alert id matches no listed alert (deleted/renamed), or None (no recoverable id — dropped, NOT routed to metricId)
for aid, ns in groups.items():
if aid in by_alert:
continue
for n in ns:
label = ("UNMATCHED alertId=%s" % aid) if aid else "UNATTRIBUTED (no alert id in targetPageRef)"
print("%s | %s" % (label, html.unescape(n.get("messageTitle", ""))))
PY
```
### Other verification paths
- **UI (per-alert trigger history)** — `/lightning/n/standard-AgentforceStudio?c__nav=alerts` → **Incidents** tab shows the full, per-alert notification list with details. Use it to corroborate the CLI attribution above or when the header path is unavailable.
- **Email** — if `deliveryConfigurations` includes `"type": "Email"`, the owner receives an email when it fires.
- **Metric value** — confirm the underlying metric actually crosses the threshold (see next section).
### Troubleshooting alerts that don't fire
First rule out the **missing-header false negative**: if the list is empty, retry with `-H "X-UNS-Type-Filter: all"` before concluding the alert never fired. If it is still empty (no notification attributed to that alert id — see attribution above), then check:
1. **Does the SDM return data?** If the semantic gateway (below) returns null/0, the alert evaluation also returns null and the condition is never met.
2. **Is the schedule running?** Alerts evaluate on their `minuteLevelFrequency`; a freshly created alert may take a few cycles.
3. **Does the threshold make sense?** A 10% rate with threshold `>= 0.5` (50%) never fires — remember the raw-0–1-ratio scale.
4. **Does the filter match live data?** An agent name in `filterContext` that never had a session selects zero rows — nothing to alert on.
---
## Verify metric value via Semantic Engine Gateway (condensed)
Use this to confirm a metric actually crosses the threshold before concluding an alert is broken. The gateway queries any `_clc` calculated measurement with time grouping / filters / aggregation.
**Endpoint:** `POST /services/data/v66.0/semantic-engine/gateway`. Inputs: SDM **record ID** (the `id`, not `apiName`), the `_clc` name (from the metric's `measurementReference.calculatedFieldApiName`), the time dimension (`AI_Agent_Session.Start_Timestamp` for session-level rates), and an ISO date range. Rate values come back as **raw 0–1 ratios**, same scale as alert thresholds.
### Single aggregate value (e.g. overall Escalation Rate for the last 24h)
```bash
body=$(mktemp) && chmod 600 "$body"
trap 'rm -f "$body"' EXIT
cat > "$body" <<'JSON'
{
"structuredSemanticQuery": {
"fields": [
{
"expression": { "semanticField": { "name": "Escalation_Rate_clc" } },
"alias": "Escalation_Rate",
"rowGrouping": false,
"semanticAggregationMethod": "SEMANTIC_AGGREGATION_METHOD_USER_AGG"
}
],
"topNFilter": { "rowsNumber": 1, "sortOrders": [] },
"options": { "limitOptions": { "limit": 2 }, "sortOrders": [], "grandTotal": false },
"semantic_context": { "currency": { "id": "" } },
"flattenFilter": {
"filters": [
{ "fieldName": "AI_Agent_Session.Start_Timestamp", "operator": "Between", "value": "2026-05-02T00:00:00.000Z|2026-05-03T00:00:00.000Z" }
],
"filterLogic": "1"
}
},
"semanticModelId": "2SMSG000000a6TV4AY"
}
JSON
sf api request rest "/services/data/v66.0/semantic-engine/gateway" \
-X POST -H "Content-Type: application/json" -b "@$body" -o <org> \
| python3 -m json.tool
```
### Time series (per-day) and agent filter
For a per-day series, add a first field with `"rowGrouping": true` and a `DATETRUNC('day', [AI_Agent_Session].[Start_Timestamp])` calculated field (alias e.g. `Day`), set `topNFilter.rowsNumber` = N, `options.limitOptions.limit` = N+1, and `grandTotal: true`. To scope to an agent, add a second `flattenFilter.filters` entry `{ "fieldName": "AI_Agent_Session_Participant.AI_Agent_Api_Name", "operator": "In", "value": "te1" }` (comma-separated for multiple) and set `"filterLogic": "1 AND 2"`.
Response rows carry `is_data_row__sl` (`true` = data row, `false` = grand total) and `grouping_1__sl` (`0` = detail, `1` = total); the `values` array is indexed by each field's `placeInOrder`.
### Discover available `_clc` measurements
```bash
SDM_API_NAME="Service_Agent_Analytics_SDM_1f8"
sf api request rest "/services/data/v66.0/ssot/semantic/models/${SDM_API_NAME}/calculated-measurements" -o <org> \
| python3 -c 'import sys,json;[print(m["apiName"],"|",m.get("label",""),"|",m.get("dataType","")) for m in json.load(sys.stdin).get("items",[])]'
```
Common Service SDM measures: `Escalation_Rate_clc`, `Deflection_Rate_clc`, `Engagement_Rate_clc`, `Abandonment_Rate_clc`, `Error_Rate_clc`, `Unique_Sessions_clc`, `Escalated_Sessions_clc`, `Average_Session_Duration_clc`.
Troubleshooting:
- **`null` values** — verify data exists in the DLO (`SELECT COUNT(*) FROM ssot__AiAgentSession__dlm` via `query-sql`). DLO has rows but gateway returns null → the SDM-to-DLO mapping is broken (provisioning issue).
- **`USER_ILLEGAL_ARGUMENT_RESOLVE_ENTITY_ERROR: Failed to resolve semantic field`** — wrong `_clc` name; list measurements with the discovery command.
- **`404 Not Found`** — confirm the path is `/services/data/v66.0/semantic-engine/gateway`; if it still 404s, retry at the org's current API version (the resource may be exposed at a different version on that org).
---
## Global troubleshooting
| Error | Cause / fix |
|---|---|
| `400 MISSING_PARAM: Owner ID cannot be empty` | The list GET requires `?ownerId={userId}`. |
| `405 Method Not Allowed` on `/tableau/dataAlerts/{id}` | Single-alert GET unsupported — list and filter by `id` client-side. |
| `401 Bad_OAuth_Token` | Token expired — `sf api request rest` refreshes automatically on the next call. |
| `404` on `/wave/dataAlerts` or `/analytics/dataAlerts` | Wrong namespace — AHM alerts live under `/tableau/dataAlerts`. |
| `FUNCTIONALITY_NOT_ENABLED: [Wave]` | Unrelated — `/tableau/dataAlerts` does not require Wave / CRM Analytics. |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |