Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/mlflow

mlflow

Verified


name: mlflow description: Expert assistance for ML lifecycle management with MLflow. Use when users need experiment tracking, model registry, LLM tracing, GenAI evaluation, prompt management, or model deployment.

MLflow - ML Lifecycle Platform

Version: 3.x | Last Updated: 2026-09-26 Python: >=3.10 | PyPI releases: 182 (latest 3.14.0 on 2026-06-17) LTS branch: 2.22.5 (2026-05-12)

Overview

MLflow is an open-source platform for managing the ML lifecycle:

  • Experiment Tracking: Log parameters, metrics, and artifacts
  • Model Registry: Version and manage models
  • LLM Tracing: Observability for LLM applications
  • GenAI Evaluation: Evaluate LLM outputs with scorers
  • Prompt Registry: Version and manage prompts

Documentation: https://mlflow.org/docs

When to Use This Skill

Activate when users need:

  • "Track ML experiments and metrics"
  • "Log and register models"
  • "Trace LLM application calls"
  • "Evaluate LLM outputs"
  • "Manage prompt versions"
  • "Deploy ML models"

Core Concepts

1. Experiment Tracking

import mlflow

# Set tracking URI and experiment
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-experiment")

# Log run
with mlflow.start_run():
    # Log parameters
    mlflow.log_params({
        "model": "gpt-4o",
        "temperature": 0.7,
        "max_tokens": 1000
    })

    # Log metrics
    mlflow.log_metric("accuracy", 0.95)
    mlflow.log_metrics({
        "precision": 0.92,
        "recall": 0.94,
        "f1": 0.93
    })

    # Log artifacts
    mlflow.log_artifact("model_config.json")
    mlflow.log_table(
        data={"prompt": prompts, "response": responses},
        artifact_file="predictions.csv"
    )

2. LLM Tracking

import mlflow

with mlflow.start_run():
    # Log LLM predictions
    mlflow.llm.log_predictions(
        inputs=[{"question": "What is MLflow?"}],
        outputs=["MLflow is an ML platform..."],
        prompts=["Answer: {input}"]
    )

3. Auto-Tracing

import mlflow

# OpenAI
mlflow.openai.autolog()

# LangChain
mlflow.langchain.autolog()

# Anthropic
mlflow.anthropic.autolog()

# LlamaIndex
mlflow.llama_index.autolog()

# All LLM calls are now traced automatically

4. Manual Tracing

import mlflow
from mlflow.entities import SpanType

@mlflow.trace(span_type=SpanType.CHAIN)
def rag_pipeline(query: str) -> str:
    """RAG pipeline with automatic tracing."""
    context = retrieve_documents(query)
    response = generate_response(query, context)
    return response

@mlflow.trace(span_type=SpanType.RETRIEVER)
def retrieve_documents(query: str) -> list:
    return vector_db.search(query, k=5)

@mlflow.trace(span_type=SpanType.LLM)
def generate_response(query: str, context: list) -> str:
    return llm.generate(query=query, context=context)

5. Context Manager Tracing

import mlflow

def complex_workflow(query: str):
    with mlflow.start_span(name="workflow") as parent_span:
        parent_span.set_inputs({"query": query})

        with mlflow.start_span(name="retrieval") as span:
            docs = retrieve(query)
            span.set_outputs({"doc_count": len(docs)})

        with mlflow.start_span(name="generation") as span:
            response = generate(query, docs)
            span.set_outputs({"response": response})

        parent_span.set_outputs({"response": response})
        return response

6. GenAI Evaluation

import mlflow
from mlflow.genai.scorers import Correctness, Guidelines
from mlflow.genai import scorer

# Define evaluation dataset
eval_dataset = [
    {
        "inputs": {"question": "What is the capital of France?"},
        "expectations": {"expected_response": "Paris"},
    },
]

# Custom scorer
@scorer
def is_concise(outputs: str) -> bool:
    return len(outputs.split()) <= 10

# Run evaluation
results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=my_model,
    scorers=[
        Correctness(),
        Guidelines(name="is_english", guidelines="Answer in English"),
        is_concise,
    ],
)

7. Predefined Scorers

from mlflow.genai.scorers import (
    Correctness,           # Validates against ground truth
    RelevanceToQuery,      # Assesses if response addresses input
    Guidelines,            # Evaluates custom criteria
    Safety,                # Detects harmful content
    Equivalence,           # Compares to expected output
    RetrievalGroundedness, # Validates RAG responses
    RetrievalRelevance,    # Ensures retrieved docs are relevant
)

scorers = [
    Correctness(),
    Guidelines(
        name="professional",
        guidelines="Response must be professional"
    ),
    Safety(),
]

8. Prompt Registry

import mlflow

# Register prompt
prompt = mlflow.genai.register_prompt(
    name="summarization-prompt",
    template="""\
Summarize in {{ num_sentences }} sentences.

Content: {{ content }}

Summary:""",
    commit_message="Initial version",
    tags={"task": "summarization"}
)

# Load prompt
prompt = mlflow.genai.load_prompt("summarization-prompt")

# Load specific version
prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt/2")

# Load by alias
prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@production")

# Format and use
formatted = prompt.format(num_sentences=3, content="...")

9. Prompt Aliases

import mlflow

# Set aliases for deployment stages
mlflow.set_prompt_alias("summarization-prompt", "production", version=3)
mlflow.set_prompt_alias("summarization-prompt", "staging", version=4)
mlflow.set_prompt_alias("summarization-prompt", "development", version=5)

# Load by alias in production
prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@production")

10. Model Registry

import mlflow

# Log model
with mlflow.start_run():
    mlflow.sklearn.log_model(
        model,
        "model",
        registered_model_name="my-classifier"
    )

# Load model
model = mlflow.sklearn.load_model("models:/my-classifier/1")

# Load latest version
model = mlflow.sklearn.load_model("models:/my-classifier/latest")

# Load by alias
model = mlflow.sklearn.load_model("models:/my-classifier@production")

Span Types

SpanTypeDescription
AGENTAgent orchestration
CHAINSequential processing
CHAT_MODELChat completion
EMBEDDINGEmbedding generation
LLMLLM inference
RETRIEVERDocument retrieval
TOOLTool/function calls
PARSEROutput parsing

Framework Integrations

FrameworkAutolog FunctionMin Version
LangChainmlflow.langchain.autolog()2.14.0
OpenAImlflow.openai.autolog()2.14.0
Anthropicmlflow.anthropic.autolog()2.17.0
LlamaIndexmlflow.llama_index.autolog()2.14.0
Geminimlflow.gemini.autolog()2.17.0
Bedrockmlflow.bedrock.autolog()2.17.0
Transformersmlflow.transformers.autolog()2.14.0
DSPymlflow.dspy.autolog()2.17.0

Common Patterns

RAG Evaluation

from mlflow.genai.scorers import RetrievalGroundedness, Correctness

results = mlflow.genai.evaluate(
    data=rag_dataset,
    predict_fn=rag_pipeline,
    scorers=[
        Correctness(),
        RetrievalGroundedness(),
    ],
)

Custom LLM Judge

from mlflow.genai.judges import make_judge
from typing import Literal

coherence_judge = make_judge(
    name="coherence",
    instructions="""
    Evaluate the coherence of the response.

    Question: {{ inputs }}
    Response: {{ outputs }}

    Rate as: coherent, somewhat coherent, or incoherent.
    """,
    feedback_value_type=Literal["coherent", "somewhat coherent", "incoherent"],
    model="openai:/gpt-4o-mini",
)

Human Feedback

import mlflow
from mlflow.entities import AssessmentSource, AssessmentSourceType

# Log human feedback
mlflow.log_feedback(
    trace_id=trace_id,
    name="helpfulness",
    value=4,
    rationale="Response was helpful",
    source=AssessmentSource(
        source_type=AssessmentSourceType.HUMAN,
        source_id="reviewer@example.com"
    )
)

Model Comparison

models = ["gpt-4o-mini", "claude-sonnet-4-20250514", "gemini-1.5-flash"]

for model in models:
    with mlflow.start_run(run_name=f"eval_{model}"):
        mlflow.log_param("model", model)
        metrics = evaluate_model(model)
        mlflow.log_metrics(metrics)

MLflow Server

# Start tracking server
mlflow server --host 0.0.0.0 --port 5000

# With database backend
mlflow server \
    --backend-store-uri postgresql://user:pass@localhost/mlflow \
    --default-artifact-root s3://mlflow-artifacts

# Start UI only
mlflow ui --port 5000

Best Practices

  1. Structured Logging: Log parameters, metrics, and artifacts consistently
  2. Experiment Organization: Use meaningful experiment names
  3. Tagging: Tag runs with environment, version, and purpose
  4. Prompt Versioning: Version prompts with meaningful commit messages
  5. Evaluation-Driven: Run evaluations on every prompt/model change

Troubleshooting

Traces Not Appearing

  • Verify autolog is enabled before making calls
  • Check tracking URI is set correctly
  • Ensure experiment is set

Missing Token Usage

  • Check MLflow version (3.1.0+ for most providers)
  • Enable stream_options={"include_usage": True} for streaming

Evaluation Failing

  • Verify scorer parameters match expected schema
  • Check predict_fn returns expected format

Resources

MLflow 3 — What's New (verified 2026-06-29, Agent 89)

  • mlflow.search_logged_models() — SQL-like filter across experiments
  • mlflow.create_external_model() — register models trained outside MLflow
  • models:/<model_id> URI — direct ID-based model loading (replaces runs:/<run_id>/path)
  • Checkpoint-aware log_metric — step=, model_id=, dataset= args
  • mlflow agent setup — installs .agents/skills for Claude Code / Codex / OpenCode
  • @mlflow.test — pytest marker for GenAI regression tests
  • Review Queues — assign traces to reviewers, collect structured feedback
  • LLM Playground — browser iteration over AI Gateway + Prompt Registry
  • WAL tracing — durable, low-latency Claude Code traces

v3.14.0 Breaking Changes

  • mlflow.sklearn serialization_format default: cloudpickle → skops (#23987)
  • mlflow.pytorch.log_model / save_model default → pt2 (#23988)
  • mlflow.lightgbm default → skops (#23986) → Pin serialization_format="cloudpickle" explicitly if you need the old behavior.

MLflow 3 — Model Checkpoint Logging

with mlflow.start_run() as run:
    for epoch in range(100):
        if epoch % 10 == 0:
            model_info = mlflow.pytorch.log_model(
                pytorch_model=model,
                name=f"checkpoint-epoch-{epoch}",
                step=epoch,
            )
            mlflow.log_metric(
                "accuracy", value, step=epoch,
                model_id=model_info.model_id,
                dataset=validation_dataset,
            )

# Load by model_id (new URI form)
loaded = mlflow.pyfunc.load_model(f"models:/{model_info.model_id}")

# Search logged models across experiments
top = mlflow.search_logged_models(
    experiment_ids=["1"],
    filter_string="metrics.accuracy > 0.9",
    order_by=[{"field_name": "metrics.accuracy", "ascending": False}],
    max_results=1,
    output_format="list",
)[0]

MLflow 3 — External Model Registration

import mlflow
mlflow.create_external_model(name="my-external-model")

Best Practices (Wave 2 additions)

  1. MLflow 3 model_id URIs: prefer models:/<model_id> over runs:/<run_id>/path for new code
  2. Serialization pinning: set serialization_format explicitly on log_model calls in v3.14.0+ to avoid the default flip
  3. Local Cianfhoghlaim code: cianfhoghlaim/core/obs/observability/mlflow_config.py should migrate runs:/{run_id}/model → models:/{model_info.model_id} for MLflow 3

Version policy

Per docs/SKILL-VERSION-HEADER-POLICY.md (the canonical refresh policy):

  • The canonical refresh cadence is weekly (Stage 7 GH workflow) + on-demand (per openspec change)
  • The canonical command is python3 scripts/audit/refresh_skill_versions.py
  • The canonical package→skill mapping is at scripts/audit/PACKAGE_TO_SKILL_MAP.md
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall