dominodatalab/domino-ai-plugin / skills/domino-trace-setup
domino-trace-setup
VerifiedSet up GenAI tracing for an LLM or agent application in Domino by checking the Domino SDK and MLflow requirements (the DSE needs nothing installed), then adding a tracing_setup.py helper, span evaluators, a config.yaml, and @add_tracing / DominoAgentContext wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.
Set Up Domino GenAI Tracing
Applies to Domino 6.3 and Domino Cloud.
Instrument an agent or LLM application so its calls, spans, and evaluator scores show up in
Domino's Experiment Manager. For traditional ML tracking, use the domino-experiment-setup
skill instead. For evaluator patterns, multi-agent examples and the full context-manager
reference, use the domino-genai-tracing skill.
Inputs
- LLM framework:
openai(default),anthropic, orlangchain. Detect from the code if possible. - Agent name: name for the traced entry-point function (letters, digits,
_and-only). - Evaluator: whether to add quality scoring, and whether to use an LLM-as-judge.
- Aggregated metrics: which evaluator metrics to summarize per run, and with which statistic.
Model access is not an input. Tracing instruments whichever client the agent already uses (a
provider SDK, a Domino-hosted endpoint, or LLM Gateway 2.0 where it is installed: domino-llm-access,
domino-llm-gateway); do not change it. Pass that client to create_llm_judge_evaluator(client=...)
so the judge shares it; through the gateway, model names are aliases.
Steps
-
Check requirements. In the Domino Standard Environment install nothing: it ships the agents code (
dominodatalab2.2.0) and MLflow 3.3.2; one harmlessFailed to end span ... '_metrics'warning per trace is expected. Do not add theagentsextra there (it downgrades MLflow sub-packages to 3.2.0). Needsdominodatalab2.1.0+ and MLflow 3.2+:python -c "import importlib.metadata as m; print({p: m.version(p) for p in ('dominodatalab', 'mlflow', 'opentelemetry-sdk')})"Outside the DSE, add to the Dockerfile instructions (or
requirements.txt), at or below the DSE versions:RUN pip install "dominodatalab[data]==2.2.0" mlflow==3.3.2 "opentelemetry-sdk<=1.44.0" "opentelemetry-api<=1.44.0" -
Copy helpers from this skill's
assets/folder next to the agent's entry point:assets/tracing_setup.py:setup_tracing(framework),create_evaluator(),create_llm_judge_evaluator(),get_aggregation_metrics(), plusinput_text()andoutput_text()for reading a span.assets/config.yaml: agent configuration logged as run parameters throughagent_config_path. Replace the model names with the ones the user actually calls; keepmodels.judgeequal to the judge model the evaluator uses.
-
Instrument the agent. Decorate the entry point with
@add_tracingand run the evaluation loop insideDominoAgentContext, as below. Keep the user's existing logic unchanged. -
Explain how to view traces (see "Viewing traces").
Example traced agent
import mlflow
from domino.agents.tracing import add_tracing
from domino.agents.logging import DominoAgentContext
from openai import OpenAI
mlflow.openai.autolog()
client = OpenAI() # the agent's existing client; a gateway or Domino-hosted-endpoint client traces the same way
def quality_evaluator(span):
"""Span evaluator: span.inputs is a dict of the call's arguments, span.outputs its return value."""
output = span.outputs or {}
scores = {"response_length": len(output.get("response", ""))}
if isinstance(output.get("confidence"), (int, float)): # report it only when the agent set it
scores["confidence"] = float(output["confidence"])
return scores
@add_tracing(name="my_agent", span_type="AGENT", evaluator=quality_evaluator)
def my_agent(query: str) -> dict:
response = client.chat.completions.create(
model="gpt-6.1-sol", # example model name; use one your provider serves
messages=[{"role": "user", "content": query}],
)
return {"response": response.choices[0].message.content, "confidence": 0.9, "model": "gpt-6.1-sol"}
if __name__ == "__main__":
aggregated_metrics = [("response_length", "mean"), ("confidence", "mean"), ("confidence", "min")]
mlflow.set_experiment("my-agent-eval") # creates the experiment if it does not exist
with DominoAgentContext(
agent_config_path="config.yaml",
custom_summary_metrics=aggregated_metrics,
) as run: # run is the mlflow ActiveRun
for query in ["What is machine learning?", "Explain neural networks"]:
result = my_agent(query)
print(f"Q: {query}\nA: {result['response'][:100]}...\n")
print(f"Run ID: {run.info.run_id}")
Two facts the example depends on; the rest of the SDK reference is in the domino-genai-tracing skill (ADD-TRACING-DECORATOR.md, DOMINO-RUN.md, EVALUATORS.md):
- An evaluator that raises records nothing (the SDK logs the error), so never return a made-up default score, and omit a metric whose input is missing. Results are tags on the shared trace, so give each decorated function's metrics distinct names.
DominoAgentContexttakesexperiment_name,run_id,agent_config_pathandcustom_summary_metricsonly (norun_name);experiment_namemust already exist, so callmlflow.set_experiment(name)first and omit it. At exit it logs one<statistic>_<metric>run metric percustom_summary_metricspair.
Framework autologging
| Framework | Setup |
|---|---|
| OpenAI | mlflow.openai.autolog() then OpenAI() |
| Anthropic | mlflow.anthropic.autolog() then Anthropic() (may need manual spans for some calls) |
| LangChain | mlflow.langchain.autolog() then e.g. ChatOpenAI(model="gpt-6.1-sol") |
@add_tracing(autolog_frameworks=["openai"]) does the same lazily on the first traced call.
Viewing traces
- Go to Experiments in the Domino project.
- Open the experiment the run logged to and select the run (an agent version).
- Open the Traces tab: one trace per call of the decorated function, with the span tree (agent, tools, LLM calls), token usage, latency and evaluator scores. Aggregated metrics are on the run.
Run the evaluation script as a Domino Job rather than from the Workspace when the run should be deployable as an agent: only Job-originated runs keep the code commit and config lineage.
Docs: https://docs.domino.ai/cloud/platform-capabilities/features/agents/develop (6.3: replace cloud with 6.3).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |