DefaultHQ/build-support-agent-skill / skills/build-support-agent
build-support-agent
VerifiedPlan and build a production customer-support agent — a RAG product-Q&A bot that drafts replies, posts them for human review, and graduates to controlled autonomy. Use when someone wants to build an in-house support agent, a Slack-native or support-platform AI bot, a ticket-answering / customer-Q&A agent, or wants to build DaanBot-style support automation. Interviews the user about scope, knowledge sources, and guardrails, then guides a phased MVP build. Triggers on 'build a support agent', 'support bot', 'AI for support tickets', 'answer customer questions automatically'.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 22.6 KB |
SKILL.md
View on GitHub---
name: build-support-agent
description: "Plan and build a production customer-support agent — a RAG product-Q&A bot that drafts replies, posts them for human review, and graduates to controlled autonomy. Use when someone wants to build an in-house support agent, a Slack-native or support-platform AI bot, a ticket-answering / customer-Q&A agent, or wants to build DaanBot-style support automation. Interviews the user about scope, knowledge sources, and guardrails, then guides a phased MVP build. Triggers on 'build a support agent', 'support bot', 'AI for support tickets', 'answer customer questions automatically'."
user-invocable: true
argument-hint: "[what you want the support agent to do]"
license: MIT
metadata:
author: Daanveer Hehr
---
# Build a Slack-Native Support Agent
These are the field notes behind **DaanBot**, a production customer-support agent
built at Default (a Series A startup) by a Forward Deployed Engineer who got tired
of typing the same answers into support threads. The public write-up is
https://www.default.com/post/building-defaults-autonomous-customer-support-agent.
This skill turns you into a
**build copilot** that helps the human ship an MVP of a similar agent, adapted to
*their* company, tools, and scope.
## How to run this skill
1. **Internalize the architecture and philosophy below.**
2. **Run the interview** (Section 5) *before writing any code*. Ask in batches,
reflect answers back, push for specificity. This is a large build and the
early decisions compound. The #1 failure mode is starting too big — push back
when the human over-scopes.
3. **Summarize what you heard** and propose a concrete MVP: one ticket type, one
channel, human-in-the-loop on every reply.
4. **Produce and execute a phased build plan** (Section 6), smallest working loop
first.
5. Keep returning to the core principle: **the model is not the special part. The
harness is** — classification, gates, retrieval quality, and guardrails are
where reliability comes from.
Do not let the human jump straight to "wire up the LLM." The first milestone is a
single deterministic loop: *ingest one message → retrieve context → draft one
reply → put it in front of a human.* Everything else is expansion.
---
## 1. What you're building
A **product Q&A assistant** that lives where the work happens — inside shared
customer channels, surfaced through a support platform. It is deliberately
*narrow*:
- **Answers product questions** ("how does X work?", "is Y supported?", "where do
I configure Z?") using a curated knowledge base.
- **Drafts** replies and posts them to an internal review channel.
- **Auto-sends** when retrieval confidence is high enough and no human has stepped
in; otherwise leaves an internal note or just a draft.
- **Escalates** to humans (with a summary + a request for diagnostic info) when it
smells a bug or hits something it shouldn't answer.
- **Stays out of the way** the moment a human is already handling the thread.
Just as important is what it is **not**: it does not access customer accounts,
make changes, run jobs, promise manual actions, or debug live systems. It never
reveals known bugs. Those boundaries are enforced in the prompt *and* in the
routing code — not left to the model's goodwill.
The framing that matters: most threads aren't "human **or** agent." They're
**"agent first, human faster."** A human still touches many threads, but they're
never starting from a blank page.
---
## 2. The architecture (the part worth copying)
The *shape* matters far more than the specific vendors:
```
Customer message (in chat, surfaced via support platform)
│
Webhook fires → your app gets a minimal payload: { issue_id }
│
1. Verify the signature — HMAC over the RAW request body bytes (captured
BEFORE any JSON.parse), compared constant-time
2. Fetch full issue + message history from the support platform's API
3. Detect: customer vs teammate — treat known bots AND the agent's own prior
messages as transparent (a "human reply" = a non-customer author NOT in
your bot/agent allowlist)
4. Detect: has a real human replied? Is the thread in a terminal state?
5. Apply gates (allow/block lists, opt-in tags, ignored channels)
6. Return 200 immediately, then process asynchronously (e.g. Next.js after(),
or a queue/worker) with a generous timeout — failures here are invisible
to the sender, so surface them yourself
│
processTicket():
│
┌─ CLASSIFY (precondition) ── cheap/fast model, structured output (schema)
│ → intent: product_question | bug_report | feature_request | billing | other
│ (product_question = the conservative default / fall-through)
│ → confidence: low | medium | high + boolean signals (hasReproSteps, …)
│ → runs BEFORE the main loop so the model can't "forget" to classify
│
├─ RETRIEVE ── embed the question → vector search → re-rank → threshold filter
│
├─ REASON ── main model + a small set of tools, multi-step (cap the steps):
│ • searchKnowledge (RAG over your docs + resolved tickets)
│ • getThreadContext (pull conversation history)
│ • flagTeam (escalate to humans with a summary)
│ • (it can also read screenshots/images attached to the thread)
│
├─ SCORE confidence ── derived from retrieval similarity, not the model's vibes
│
└─ ACT (decided by routing CODE, not the model):
• no human yet + confidence not low → public auto-reply
• blocked account / human already in → internal note only
• low confidence / ineligible → draft only, wait for a human
│
Post draft to the review channel → humans approve/edit/reject via emoji
│
Reactions feed a feedback table → future eval + training signal
```
### Six design decisions that carried the project
1. **Classification is a precondition, not a tool.** A separate structured-output
call (cheap model + a schema) runs *before* the main reasoning loop. Its output
is canonical — the main prompt is told not to re-classify. Predictable,
debuggable behavior.
2. **Routing decisions live in code, not in the prompt.** Whether to reply
publicly, leave a note, or stay silent is decided by deterministic gates
(human-replied? terminal state? opt-in tag? blocked account?). The model drafts
text; it does not decide whether that text reaches a customer.
3. **Confidence comes from retrieval, not self-report.** A weighted blend of the
top and average similarity of retrieved sources (e.g. `0.6*top + 0.4*avg`).
Models are bad at knowing when they're wrong; retrieval scores are more honest.
4. **Fail soft on everything optional.** Classifier errors → fall back to
`product_question / low` and keep going. DB creds missing → skip the write,
don't crash. The core loop is never blocked by a non-essential dependency.
5. **Dependency injection at one boundary.** No package reads environment
variables. All clients (LLM provider, vector DB, support platform, chat, DB)
are constructed in one place and passed down as typed interfaces. This is what
makes it testable and swappable.
6. **Idempotency lives in a durable store, not process memory.** Webhook senders
retry and customers send bursts, so you *will* get the same event more than
once. Dedup on a stable key — `(issue_id + latest_customer_message_id)` — in a
durable store (a DB row with a unique constraint, or Redis with a TTL). Do not
use an in-process Map on serverless: it does not survive cold starts and is not
shared across concurrent instances. We shipped an in-memory version first and it
silently failed to dedup in production.
### The phased rollout (do not skip the phases)
- **Phase 0 — MVP:** one ticket type (product questions), RAG search, draft to a
review channel, a human sends every reply. No auto-send, no classification.
- **Phase 1 — Classify in shadow mode:** add the classifier, log its output on
every ticket, gate *nothing* on it. Watch precision for days before trusting it.
- **Phase 2 — Default silence + opt-in:** the agent goes quiet once a human
engages; humans re-invite it with tags/commands. This made it safe to run in
real customer channels.
- **Phase 3 — Escalation routing (next):** high-confidence bug → acknowledge, move
to a "bug preview" state, flag the team, stop answering it like a Q&A. Feature
request → route it. Billing → billing path.
Each phase shipped only after the previous one was *solid*. There was always a
single loop to debug.
---
## 3. The stack — choices and **honest** rationale
Be honest with the human about the bias: **most of these were chosen because they
were already the tools the company used internally.** That's a legitimate,
underrated reason — hooking into systems your team already lives in means less
integration surface, less new auth, and an agent that sits where the humans
already are. Optimize for the *existing* stack before chasing the "best" tool.
| Layer | Reference choice | Why (honest version) | Strong alternatives |
|---|---|---|---|
| **Reasoning model** | Anthropic Claude (capable model for synthesis, cheap/fast model for classification) | Strong instruction-following; the cheap tier is good enough for classification. Tiering saves real money. | OpenAI (GPT for synthesis, mini for classification), Google Gemini, open models via Together/Fireworks/Groq, or a model gateway to swap freely |
| **Agent framework** | Vercel AI SDK | Made tool-calling, structured output, and multi-step loops trivial; provider-agnostic. | LangChain/LangGraph, Mastra, provider SDKs directly, Pydantic AI (Python), or raw API calls for a simple loop |
| **Embeddings** | OpenAI `text-embedding-3-large` | Strong retrieval quality, dead-simple API. | Cohere, Voyage AI, Google, open models (bge/e5/nomic) self-hosted |
| **Vector DB** | Pinecone | Managed, fast, zero ops, good metadata filtering. | pgvector (if you already run Postgres — often the right call), Weaviate, Qdrant, Turbopuffer, Milvus, Chroma (local/dev) |
| **Support platform** | Pylon | System of record; clean REST API; native shared-channel support. | Plain, Zendesk, Intercom, Front, Help Scout, Linear (issues), or a thin layer over the chat API directly |
| **Human review surface** | Slack | Team already lives there; emoji reactions are a zero-friction feedback UI. | Microsoft Teams, Discord, a lightweight internal dashboard, email |
| **Database** | Supabase (Postgres) | Managed Postgres + instant API. | Plain Postgres (Neon/RDS/Railway), PlanetScale, SQLite/Turso at small scale, Firebase |
| **App hosting** | Vercel | Web app + serverless webhook handler; trivial deploys. | Render, Railway, Fly.io, Cloudflare Workers, AWS Lambda + API Gateway, plain Node server |
| **Long-running listener** | Railway | A socket-mode reaction listener needs a persistent process serverless can't host. | Fly.io, Render worker, a small VM, or use the chat platform's Events API webhooks instead (then no long-running process needed) |
| **Secrets** | Doppler | One source of truth synced to every env. | Host-native env vars, AWS Secrets Manager, 1Password, Infisical, plain `.env` for a prototype |
| **Monorepo tooling** | pnpm workspaces + Turborepo | Clean package boundaries, fast cached builds. | npm/yarn workspaces, Nx, Bun workspaces, or a single package if you don't need the split |
> **Pin your agent-framework major version up front.** The reference build is
> Vercel AI SDK v4 (`ai@^4`, `@ai-sdk/anthropic@^1`, Zod 3): multi-step is a numeric
> `maxSteps` and structured output is `generateObject`. v5+ replaces `maxSteps` with
> `stopWhen: stepCountIs(n)` and ships `@ai-sdk/anthropic` v2 with different
> message/part shapes — copied snippets won't compile across the boundary. Pick one
> major and keep the app and all packages aligned (a pnpm catalog helps). The same
> "pin the major" caution applies to whichever framework you choose.
**The meta-point:** components matter far less than their assembly. A generic
support agent fails not because it picked the wrong vector DB, but because it has
no support history, no product knowledge, no real questions, no screenshots, and
no domain guardrails. The moat is the context and the harness, not the SKU.
---
## 4. Non-obvious lessons (share these with the human)
- **Start with one repeated, mostly-deterministic task and refuse to expand until
it's solid.** Product Q&A is ideal: high volume, the answer usually already
exists, and the hard part is *retrieval + clear phrasing*, not deciding what to
do. If you do the same thing over and over, there's a process buried in there
with a shape you can codify.
- **Most AI projects fail by doing everything at once.** No single loop → no loop
to debug. Narrow scope is a feature.
- **The model isn't the special part — the harness is.** Reliability came from
classification logic, behavior gates, and tightly-scoped tools.
- **Images matter more than you'd expect.** Customers send screenshots constantly
(errors, config, workflow states). But don't hand the attachment URL to the
model: support/chat platforms hand back short-lived, pre-signed CDN URLs the
model provider's servers usually can't fetch. Download each image server-side
into a buffer and pass it inline as image bytes + mime type. Cap the size (e.g.
5MB) by checking *both* the content-length header and the actual byte length
after download, and skip non-image content-types.
- **Give the agent your product's navigation paths** (slugs/routes) so it
describes the actual flow, not a generic "go to settings."
- **Guardrails belong in code *and* prompt:** never reveal known bugs, never
promise manual actions, never fabricate URLs/integrations/paths, never leak
internal-note formatting into customer replies.
- **Cost-tier your models.** Classification runs on *every* ticket — make it
cheap. Synthesis runs on a capable model.
- **Ship the feedback loop early, even if it's just emoji.** Approve/edit/reject
reactions become your eval set and future training data.
- **Make bots and the agent itself transparent in "has a human replied?" logic.**
Keep an allowlist of bot/system user ids — *including your own agent's* — and
treat their messages as non-human. Otherwise the platform's auto-responder or
your agent's own last reply fools it into going silent. Keep a separate guard so
it never replies twice to the same customer turn.
- **Expect message bursts.** Customers fire several messages in seconds. A short
tail-debounce that coalesces rapid messages and re-fetches the full thread before
drafting lets the agent answer the complete context once instead of replying to
each fragment — back it with the durable store from decision 6, not an in-memory
timer.
---
## 5. THE INTERVIEW — run this before writing code
Ask in batches. Reflect answers back. Push for specificity. Nail **scope,
context sources, and guardrails** — the three things that decide whether this
works. Don't move on until you can name the *one* loop you'll build first.
### A. Scope & the first loop
1. What single, repetitive support task do you do over and over? Describe the last
3 times you did it.
2. Roughly what fraction of your ticket volume is "the answer already exists
somewhere"? (That's your candidate MVP surface.)
3. What's the *one* ticket type for the MVP? (Strongly bias toward product Q&A.)
4. What is explicitly **out of scope** for v1? (Force them to name things.)
5. What must the agent **never** do? (Account access? Promises? Refunds? Mention
bugs?) These become hard-coded guardrails.
### B. Where the work happens
6. Where do customers ask questions today — Slack, email, in-app chat, a portal?
Shared channels or 1:1?
7. What support platform (if any) is your system of record? Does it have a REST
API and outbound webhooks? Can you get message history programmatically?
8. Do customers send screenshots/images often? (If yes, plan for vision.)
9. Where does your team want to *review* drafts — Slack, Teams, a dashboard?
### C. Knowledge & context (the moat)
10. What sources hold the answers today? (Docs, help center, Notion, resolved
tickets, chat history, engineers' heads?)
11. How much is written down vs. tribal knowledge?
12. Do you have a corpus of *resolved* tickets/past replies to mine? (Gold — it
teaches tone and real answers.)
13. Are there product navigation paths/routes/slugs you could feed the agent?
14. How fresh must the knowledge be? How often do answers change after a release?
15. Any PII/sensitive data that must be redacted before indexing?
### D. Behavior & autonomy
16. MVP: human-in-the-loop on *every* reply (recommended), or auto-send on high
confidence from day one? (Recommend the former.)
17. What happens when a human is already handling a thread — go silent?
18. How should it escalate when it can't answer or smells a bug — who gets pinged,
what info should it collect from the customer?
19. Want an opt-in mechanism (tags/commands) for humans to re-invite the agent?
20. What ticket categories do you want to classify into eventually? (taxonomy)
### E. Stack & constraints
21. What's your existing stack? (Language, framework, hosting, DB.) **Lean on what
you already run** — name it.
22. Which LLM provider(s) can you use? Any data-residency/vendor constraints?
23. Do you already run Postgres? (If yes, pgvector may beat a new vector DB.)
24. Hosting: serverless OK, or do you need a long-running process for the chat
listener?
25. How do you manage secrets today?
26. Budget sensitivity — rough monthly ticket volume? (Drives model-tier choices.)
### F. Success & iteration
27. How will you know v1 works? Define a concrete metric (e.g. "% of
product-question drafts the team sends with zero edits").
28. Who reviews drafts, and how do they give feedback (emoji, buttons, a form)?
29. Weekend prototype or quarter-long build?
30. After the MVP works, what's the *next* single capability — and why not before?
> If the human answers Q3 with more than one ticket type, or answers Q4 with
> "nothing," **stop and re-scope.** That's the failure pattern.
---
## 6. The MVP build plan (your template after the interview)
Adapt to their answers. Build the smallest working loop first; each milestone is
independently demoable.
**Milestone 1 — One deterministic loop (no LLM judgment yet).**
Receive a message (webhook, poll, or manual trigger). Verify the signature by
computing the HMAC over the RAW request body bytes, captured *before* JSON.parse —
re-stringifying the parsed object changes whitespace and key order and breaks the
hash. Compare with a timing-safe check and fail closed (missing secret → 500,
missing/invalid signature → 401); a debug bypass is fine in dev but never in prod.
Acknowledge fast, process after: senders retry slow or failed responses (causing
duplicate fires), so return 200 immediately and do the heavy work in a deferred
task (Next.js `after()`, or a queue/worker) with a generous timeout. Because you
already returned 200, any error in that deferred work is invisible to the sender —
from day one, post processing failures to your review channel (a one-line
"⚠️ Agent processing failed" with ticket id + error) so a crash surfaces where a
human will see it. Fetch the full thread, hardcode "always draft, never send," and
post a placeholder draft to the review channel. Prove the plumbing end-to-end
before any AI.
**Milestone 2 — Retrieval.**
Pick 20–50 of the best existing answers/docs. Redact PII. Embed and index them. On
a new message: embed the question, retrieve top-K, show sources in the draft.
Eyeball retrieval quality on real questions before going further.
**Milestone 3 — Drafting.**
Add the synthesis model + a `searchKnowledge` tool + a tight system prompt (tone,
hard guardrails, "only cite retrieved sources, no fabricated URLs"). Compute a
confidence score from retrieval similarity; show it on every draft. Still
human-in-the-loop: every reply is sent by a person.
**Milestone 4 — Feedback loop.**
Add approve/edit/reject reactions in the review channel. Persist them. This is your
eval set — watch where it gets edited and why.
**Milestone 5 — Classification in shadow mode.**
Add the cheap-model classifier (structured output, schema). Run it deterministically
(temperature 0) and raise the provider retry count above the SDK default — provider
"overloaded" spikes outlast the usual two retries (the reference uses five). Use a
small intent set with `product_question` as the conservative default / fall-through
(when it's a question but not clearly a bug, FR, or billing). Log intent +
confidence on every ticket. Gate nothing yet. Measure precision for days.
**Milestone 6 — Controlled autonomy.**
*Now* allow auto-send on a narrow slice: high retrieval confidence, no human yet,
product-question intent. Everything else stays draft-only. Add "go silent once a
human engages" + an opt-in tag to re-invite the agent. Before auto-send goes live,
add the durable idempotency key from design decision 6 — a duplicated webhook now
means a duplicated *customer* reply.
**Then, and only then**, discuss escalation routing, image understanding,
navigation-path knowledge, and a feedback-to-knowledge pipeline.
### Guardrails to bake in from the start
- Never reveal known bugs or past incidents to customers.
- Never promise actions the team hasn't committed to.
- Only cite URLs/integrations/navigation paths from retrieval or an allowlist.
- Strip internal-note formatting from anything customer-visible.
- Sanitize and length-cap customer input; log (don't necessarily block) obvious
prompt-injection patterns.
- Fail soft: a broken optional dependency must not take down the core loop.
---
## 7. Where this is headed (motivation)
The agent today classifies and assists. Next it *moves tickets onto the right
path*: a high-confidence bug gets acknowledged, moved to a bug-preview state, and
flagged to the team instead of answered like a Q&A. The version after that is the
exciting one: the support agent as the **front door to a deeper internal agent
system** — reads the message and screenshots, searches recent product changes,
reproduces the bug in a sandbox, files a tracked ticket with a trace, and drafts a
PR for an engineer to review.
That works because the foundation was built for it: a support agent is really just
a **workflow** — inputs, retrieval, classification, tools, guardrails, outputs,
and human checkpoints. Repeated human processes have shapes. Write the shape down,
wrap it in context and guardrails, and you have a playbook a machine can follow.
**Start with one task. Strong context. A clear edge. Earn the right to more.**
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |