Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/mlflow

mlflow

Verified

references/mlflow-llm-guide.md

View on GitHub
# MLflow LLM Features Reference Documentation

> Comprehensive guide to MLflow's LLM-specific features for tracking, evaluating, and managing Large Language Model applications.

## Table of Contents

1. [MLflow LLM Tracking](#1-mlflow-llm-tracking)
2. [MLflow Evaluate for LLMs](#2-mlflow-evaluate-for-llms)
3. [MLflow Tracing](#3-mlflow-tracing)
4. [Prompt Engineering](#4-prompt-engineering)
5. [RAG Support](#5-rag-retrieval-augmented-generation-support)
6. [Best Practices](#6-best-practices)

---

## 1. MLflow LLM Tracking

MLflow's LLM Tracking component provides APIs for logging inputs, outputs, and prompts from LLM interactions, along with a UI for viewing experimental results.

### Core Concepts

- **Runs**: Each run represents a distinct execution or interaction with the LLM
- **Predictions**: Encompasses prompts/inputs and outputs/responses, stored as CSV artifacts
- **Parameters**: Key-value input parameters (temperature, top_k, etc.)
- **Metrics**: Quantitative insights (accuracy, response time, etc.)
- **Artifacts**: Output files including models, visualizations, and data files

### mlflow.llm Module API

#### mlflow.llm.log_predictions()

Logs a batch of inputs, outputs, and prompts for the current evaluation run.

```python
mlflow.llm.log_predictions(
    inputs: List[Union[str, Dict[str, str]]],
    outputs: List[str],
    prompts: List[Union[str, Dict[str, str]]]
) -> None
```

**Parameters:**
- `inputs`: List of input strings or input dictionaries
- `outputs`: List of output strings
- `prompts`: List of prompt strings or prompt dictionaries

**Example:**

```python
import mlflow

inputs = [
    {
        "question": "How do I create a Databricks cluster with UC access?",
        "context": "Databricks clusters are...",
    },
]
outputs = [
    "<Instructions for cluster creation with UC enabled>",
]
prompts = [
    "Get Databricks documentation to answer all the questions: {input}",
]

with mlflow.start_run():
    # Log LLM predictions
    mlflow.llm.log_predictions(inputs, outputs, prompts)
```

### Logging Parameters and Metrics

```python
import mlflow

with mlflow.start_run():
    # Log LLM parameters
    mlflow.log_params({
        "model": "gpt-4",
        "temperature": 0.7,
        "top_k": 50,
        "max_tokens": 1000
    })

    # Log metrics
    mlflow.log_metric("response_time_ms", 245)
    mlflow.log_metric("token_count", 150)
    mlflow.log_metric("accuracy", 0.92)

    # Log multiple metrics at once
    mlflow.log_metrics({
        "precision": 0.89,
        "recall": 0.91,
        "f1_score": 0.90
    })
```

### Logging Artifacts

```python
import mlflow

with mlflow.start_run():
    # Log individual artifact
    mlflow.log_artifact("model_config.json")

    # Log table data (CSV format)
    mlflow.log_table(
        data={"prompt": prompts, "response": responses},
        artifact_file="predictions.csv"
    )
```

---

## 2. MLflow Evaluate for LLMs

MLflow provides comprehensive evaluation capabilities for LLMs using both traditional metrics and LLM-as-a-Judge approaches.

### Modern GenAI Evaluation API (MLflow 3.x)

#### mlflow.genai.evaluate()

The modern evaluation function for GenAI applications.

```python
import mlflow
from mlflow.genai.scorers import Correctness, Guidelines
from mlflow.genai import scorer
from openai import OpenAI

# Set up MLflow
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("GenAI Evaluation")

# Create prediction function
client = OpenAI()
def qa_predict_fn(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer questions concisely."},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

# Define evaluation dataset
eval_dataset = [
    {
        "inputs": {"question": "What is the capital of France?"},
        "expectations": {"expected_response": "Paris"},
    },
    {
        "inputs": {"question": "Who wrote Romeo and Juliet?"},
        "expectations": {"expected_response": "William Shakespeare"},
    },
]

# Define custom scorer
@scorer
def is_concise(outputs: str) -> bool:
    """Check if answer is concise (less than 5 words)"""
    return len(outputs.split()) <= 5

# Run evaluation
results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=qa_predict_fn,
    scorers=[
        Correctness(),
        Guidelines(name="is_english", guidelines="The answer must be in English"),
        is_concise,
    ],
)
```

### Predefined LLM Scorers

MLflow provides 8 built-in scorers for common evaluation needs:

| Scorer | Purpose | Required Data |
|--------|---------|---------------|
| `Correctness` | Validates responses against ground truth | expectations |
| `RelevanceToQuery` | Assesses if responses address user input | inputs |
| `Guidelines` | Evaluates adherence to custom criteria | guidelines |
| `ExpectationsGuidelines` | Checks if responses meet specific expectations | expectations |
| `Safety` | Detects harmful or toxic content | outputs |
| `Equivalence` | Compares responses to expected outputs | expectations |
| `RetrievalGroundedness` | Validates responses use retrieved info | trace |
| `RetrievalRelevance` | Ensures retrieved docs match requests | trace |

**Usage with Different Models:**

```python
from mlflow.genai.scorers import Correctness, RelevanceToQuery, Guidelines

# Use different LLM providers as judges
scorers = [
    Correctness(model="openai:/gpt-4o-mini"),
    Correctness(model="anthropic:/claude-sonnet-4-20250514"),
    Correctness(model="google:/gemini-2.0-flash"),
    RelevanceToQuery(),
    Guidelines(
        name="professional_tone",
        guidelines="Response must maintain a professional and respectful tone"
    ),
]
```

### Custom Code-Based Scorers

Create custom evaluation logic using the `@scorer` decorator:

```python
from mlflow.genai import scorer
from mlflow.entities import Feedback, AssessmentSource, SpanType

# Simple boolean scorer
@scorer
def exact_match(outputs: dict, expectations: dict) -> bool:
    return outputs == expectations["expected_response"]

# Numeric scorer
@scorer
def word_count_score(outputs: str) -> float:
    """Score based on response length"""
    count = len(outputs.split())
    return min(count / 100, 1.0)  # Normalize to 0-1

# Scorer with rich feedback
@scorer
def content_quality(outputs: str) -> Feedback:
    score = calculate_quality(outputs)
    return Feedback(
        value=score,
        rationale="Assessment based on clarity, accuracy, and completeness",
        source=AssessmentSource(source_type="CODE", source_id="v1.0"),
        metadata={"scorer_version": "1.0", "model": "custom"}
    )

# Trace-based scorer for agent evaluation
@scorer
def uses_correct_tools(trace, expectations: dict) -> Feedback:
    """Check if agent used the expected tools"""
    tool_spans = trace.search_spans(span_type=SpanType.TOOL)
    used_tools = {span.name for span in tool_spans}
    expected_tools = set(expectations.get("expected_tools", []))

    correct = expected_tools.issubset(used_tools)
    return Feedback(
        value=correct,
        rationale=f"Used tools: {used_tools}, Expected: {expected_tools}"
    )
```

**Scorer Parameter Options:**

```python
@scorer
def my_scorer(
    *,
    inputs: dict[str, Any],       # Input data/questions
    outputs: Any,                  # Generated responses
    expectations: dict[str, Any],  # Ground truth
    trace: Trace,                  # Full execution trace
) -> float | bool | str | Feedback | list[Feedback]:
    pass
```

### Custom LLM Judge Metrics

#### make_judge() - Modern API (MLflow >= 3.4.0)

```python
from mlflow.genai.judges import make_judge
from typing import Literal

# Create coherence judge
coherence_judge = make_judge(
    name="coherence",
    instructions="""
    Evaluate the coherence of the response.

    Question: {{ inputs }}
    Response: {{ outputs }}

    Rate the coherence as: coherent, somewhat coherent, or incoherent.
    Consider logical flow, consistency, and clarity.
    """,
    feedback_value_type=Literal["coherent", "somewhat coherent", "incoherent"],
    model="openai:/gpt-4o-mini",
)

# Use in evaluation
results = mlflow.genai.evaluate(
    data=test_data,
    predict_fn=my_model,
    scorers=[coherence_judge],
)
```

**Template Variables for make_judge:**
- `{{ inputs }}` - Input data/questions
- `{{ outputs }}` - Generated responses
- `{{ expectations }}` - Ground truth
- `{{ trace }}` - Full agent trace data

#### make_genai_metric() - Legacy API

```python
from mlflow.metrics.genai import make_genai_metric, EvaluationExample

# Create evaluation examples
professionalism_example = EvaluationExample(
    input="What is MLflow?",
    output="MLflow is an open-source platform for managing the ML lifecycle.",
    score=5,
    justification="Professional tone, clear and concise explanation."
)

# Create custom metric
professionalism_metric = make_genai_metric(
    name="professionalism",
    definition=(
        "Professionalism refers to the use of a formal, respectful, and "
        "appropriate style of communication."
    ),
    grading_prompt=(
        "Professionalism scoring criteria:\n"
        "- Score 1: Extremely casual, includes slang\n"
        "- Score 2: Somewhat casual\n"
        "- Score 3: Neutral tone\n"
        "- Score 4: Mostly professional\n"
        "- Score 5: Highly professional and polished"
    ),
    examples=[professionalism_example],
    model="openai:/gpt-4",
    parameters={"temperature": 0.0},
)

# Use in evaluation
results = mlflow.evaluate(
    model=my_model,
    data=eval_data,
    extra_metrics=[professionalism_metric],
)
```

### Evaluation Datasets

#### Creating Managed Datasets

```python
from mlflow.genai.datasets import create_dataset

# Create a new evaluation dataset
dataset = create_dataset(
    name="customer_support_qa_v1",
    experiment_id=["0"],
    tags={
        "version": "1.0",
        "purpose": "regression_testing",
        "model": "gpt-4",
        "team": "ml-platform",
    },
)
```

#### Adding Records to Datasets

```python
# From trace data with expectations
traces = mlflow.search_traces(
    experiment_ids=["0"],
    max_results=50,
    filter_string="attributes.name = 'chat_completion'",
    return_type="list",
)

# Add expectations to traces
for trace in traces[:20]:
    mlflow.log_expectation(
        trace_id=trace.info.trace_id,
        name="output_quality",
        value={"relevance": 0.95, "accuracy": 1.0},
    )
```

#### Using Datasets in Evaluation

```python
from mlflow.genai import evaluate
from mlflow.genai.scorers import Correctness, Guidelines

results = evaluate(
    data=dataset,
    predict_fn=my_model.predict,
    scorers=[
        Correctness(name="factual_accuracy"),
        Guidelines(
            name="support_quality",
            guidelines="Response must be helpful, accurate, and professional",
        ),
    ],
)
```

### Human Feedback and Assessments

#### Logging Expectations (Ground Truth)

```python
import mlflow
from mlflow.entities import AssessmentSource, AssessmentSourceType

# Log ground truth expectation
mlflow.log_expectation(
    trace_id="tr-1234567890abcdef",
    name="correct_answer",
    value={"expected": "Paris is the capital of France"},
    source=AssessmentSource(
        source_type=AssessmentSourceType.HUMAN,
        source_id="expert@example.com"
    ),
    metadata={"annotator_confidence": "high"}
)
```

#### Logging Feedback

```python
# Log human feedback
mlflow.log_feedback(
    trace_id=trace_id,
    name="helpfulness",
    value=4,  # 1-5 scale
    rationale="Response was helpful but could be more detailed",
    source=AssessmentSource(
        source_type=AssessmentSourceType.HUMAN,
        source_id="reviewer@example.com"
    )
)

# Log automated feedback
mlflow.log_feedback(
    trace_id=trace_id,
    name="syntax_valid",
    value=True,
    source=AssessmentSource(
        source_type=AssessmentSourceType.CODE,
        source_id="syntax_checker_v1"
    )
)

# Log LLM judge feedback
mlflow.log_feedback(
    trace_id=trace_id,
    name="coherence_score",
    value=0.85,
    rationale="Response maintains logical flow throughout",
    source=AssessmentSource(
        source_type=AssessmentSourceType.LLM_JUDGE,
        source_id="gpt-4-judge"
    )
)
```

---

## 3. MLflow Tracing

MLflow Tracing provides OpenTelemetry-compatible observability for LLM applications, capturing inputs, outputs, and metadata for debugging and monitoring.

### Auto-Tracing

Enable automatic tracing with a single line for supported frameworks:

```python
import mlflow

# OpenAI
mlflow.openai.autolog()

# LangChain
mlflow.langchain.autolog()

# LlamaIndex
mlflow.llama_index.autolog()

# Anthropic
mlflow.anthropic.autolog()

# And many more...
```

**Supported Frameworks (28+):**

| Category | Frameworks |
|----------|------------|
| Agent Platforms | LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agents, PydanticAI, DSPy |
| LLM Providers | OpenAI, Anthropic, Google Gemini, AWS Bedrock, Groq, Mistral, Ollama |
| RAG Frameworks | LlamaIndex, Haystack, txtai |
| Other | Instructor, LiteLLM, Vercel AI SDK |

### Manual Tracing

#### Using the @mlflow.trace Decorator

```python
import mlflow
from mlflow.entities import SpanType

@mlflow.trace(span_type=SpanType.CHAIN)
def process_query(query: str) -> str:
    """Process a user query through multiple steps"""

    # Retrieve context
    context = retrieve_documents(query)

    # Generate response
    response = generate_response(query, context)

    return response

@mlflow.trace(span_type=SpanType.RETRIEVER)
def retrieve_documents(query: str) -> list:
    """Retrieve relevant documents"""
    return vector_db.search(query, k=5)

@mlflow.trace(span_type=SpanType.LLM)
def generate_response(query: str, context: list) -> str:
    """Generate response using LLM"""
    return llm.generate(query=query, context=context)
```

**Decorator Parameters:**

```python
@mlflow.trace(
    name="custom_name",           # Override default span name
    span_type=SpanType.LLM,       # Set span type
    attributes={"version": "1.0"} # Add custom metadata
)
def my_function():
    pass
```

#### Using Context Manager

```python
import mlflow

def complex_workflow(query: str):
    with mlflow.start_span(name="workflow") as parent_span:
        parent_span.set_inputs({"query": query})

        # First step
        with mlflow.start_span(name="retrieval") as retrieval_span:
            docs = retrieve(query)
            retrieval_span.set_outputs({"doc_count": len(docs)})

        # Second step
        with mlflow.start_span(name="generation") as gen_span:
            gen_span.set_inputs({"query": query, "context": docs})
            response = generate(query, docs)
            gen_span.set_outputs({"response": response})

        parent_span.set_outputs({"response": response})
        return response
```

#### Combining Auto-Tracing with Manual Spans

```python
import mlflow
from mlflow.entities import SpanType

# Enable auto-tracing for OpenAI
mlflow.openai.autolog()

@mlflow.trace(span_type=SpanType.CHAIN)
def rag_pipeline(question: str) -> str:
    """RAG pipeline with auto-traced LLM calls"""

    # This retrieval is manually traced
    with mlflow.start_span(name="custom_retrieval") as span:
        docs = custom_retriever.search(question)
        span.set_outputs({"retrieved_docs": len(docs)})

    # OpenAI call is automatically traced
    response = openai_client.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": f"Context: {docs}"},
            {"role": "user", "content": question}
        ]
    )

    return response.choices[0].message.content
```

### Span Types

MLflow provides predefined span types for categorizing different operations:

| SpanType | Description |
|----------|-------------|
| `AGENT` | Agent orchestration |
| `CHAIN` | Sequential processing chain |
| `CHAT_MODEL` | Chat completion API |
| `EMBEDDING` | Embedding generation |
| `LLM` | LLM inference |
| `MEMORY` | Memory operations |
| `PARSER` | Output parsing |
| `RERANKER` | Result reranking |
| `RETRIEVER` | Document retrieval |
| `TOOL` | Tool/function calls |

```python
from mlflow.entities import SpanType

# Custom span type (string)
@mlflow.trace(span_type="ROUTER")
def route_query(query):
    pass
```

### Trace Utilities

```python
import mlflow

# Get current active span
span = mlflow.get_current_active_span()
if span:
    span.set_attribute("custom_metric", 0.95)

# Update current trace with metadata
mlflow.update_current_trace(
    tags={"environment": "production"},
    request_preview="User query about...",
    response_preview="Response summary..."
)

# Search traces
traces = mlflow.search_traces(
    experiment_ids=["0"],
    filter_string="attributes.name = 'rag_pipeline'",
    max_results=100
)
```

### Production Tracing Configuration

```python
import mlflow

# Enable async logging for production
mlflow.config.enable_async_logging(True)

# Use lightweight tracing SDK
# pip install mlflow-tracing  # Minimal dependencies

# Disable tracing when needed
mlflow.tracing.disable()

# Re-enable tracing
mlflow.tracing.enable()
```

---

## 4. Prompt Engineering

MLflow Prompt Registry provides centralized management for prompt templates with versioning, aliasing, and collaboration features.

### Core Concepts

- **Prompt**: Versioned template with variables in `{{variable}}` format
- **Version**: Immutable, sequential revision numbers
- **Alias**: Mutable references (e.g., "production", "staging") for deployment
- **Tags**: Metadata for organization and filtering

### Creating and Registering Prompts

```python
import mlflow

# Register a new prompt
initial_template = """\
Summarize the following content in {{ num_sentences }} sentences.

Content: {{ content }}

Summary:"""

prompt = mlflow.genai.register_prompt(
    name="summarization-prompt",
    template=initial_template,
    commit_message="Initial version",
    tags={
        "author": "author@example.com",
        "task": "summarization",
        "language": "en",
    },
)

print(f"Created prompt: {prompt.name}, version: {prompt.version}")
```

### Creating New Versions

```python
# Update the prompt (creates new version)
updated_template = """\
Summarize the following content in {{ num_sentences }} sentences.
Focus on key points and maintain clarity.

Content: {{ content }}

Summary:"""

updated_prompt = mlflow.genai.register_prompt(
    name="summarization-prompt",
    template=updated_template,
    commit_message="Added focus instruction for better quality",
    tags={"author": "author@example.com"},
)

print(f"New version: {updated_prompt.version}")
```

### Loading Prompts

```python
import mlflow

# Load latest version
prompt = mlflow.genai.load_prompt("summarization-prompt")

# Load specific version
prompt_v1 = mlflow.genai.load_prompt("prompts:/summarization-prompt/1")

# Load by alias
prod_prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@production")

# Use the @latest reserved alias
latest = mlflow.genai.load_prompt("prompts:/summarization-prompt@latest")
```

### Managing Aliases

```python
import mlflow

# Set alias for deployment
mlflow.set_prompt_alias(
    name="summarization-prompt",
    alias="production",
    version=2
)

mlflow.set_prompt_alias(
    name="summarization-prompt",
    alias="staging",
    version=3
)

# Load by alias in application code
prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@production")
```

### Using Prompts in Applications

```python
import mlflow
from openai import OpenAI

# Load prompt
prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@production")

# Format with variables
formatted = prompt.format(
    num_sentences=3,
    content="MLflow is an open-source platform..."
)

# Use with LLM
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": formatted}]
)
```

### Integration with LangChain

```python
import mlflow
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

# Load from MLflow
mlflow_prompt = mlflow.genai.load_prompt("summarization-prompt")

# Convert to LangChain format
langchain_template = PromptTemplate.from_template(mlflow_prompt.template)

# Use in chain
chain = langchain_template | ChatOpenAI(model="gpt-4")
result = chain.invoke({
    "num_sentences": 3,
    "content": "Content to summarize..."
})
```

### Best Practices for Prompt Management

1. **Descriptive Naming**: Use clear names reflecting purpose (e.g., `customer-support-qa`, `code-review-feedback`)
2. **Meaningful Commits**: Document changes in commit messages
3. **Logical Tagging**: Use tags for categorization (task type, language, model)
4. **Alias Strategy**: Use environment-based aliases (production, staging, development)
5. **Version Immutability**: Never try to modify existing versions; create new ones

---

## 5. RAG (Retrieval-Augmented Generation) Support

MLflow provides specialized evaluation capabilities for RAG systems, including retrieval metrics and context relevance scoring.

### RAG Evaluation Metrics

#### Built-in Retriever Metrics

```python
import mlflow

# Evaluate retriever performance
results = mlflow.evaluate(
    model=retriever_model,
    data=eval_data,
    targets="ground_truth_docs",
    model_type="retriever",
    evaluators="default",
    evaluator_config={
        "k": [1, 3, 5, 10],  # Evaluate at different k values
    }
)

# Metrics include:
# - precision_at_k
# - recall_at_k
# - ndcg_at_k
```

#### RAG-Specific Scorers

```python
from mlflow.genai.scorers import RetrievalGroundedness, RetrievalRelevance

results = mlflow.genai.evaluate(
    data=rag_dataset,
    predict_fn=rag_pipeline,
    scorers=[
        RetrievalGroundedness(),  # Is response grounded in retrieved docs?
        RetrievalRelevance(),     # Are retrieved docs relevant to query?
    ],
)
```

### Evaluating Complete RAG Systems

```python
import mlflow
from mlflow.genai.scorers import Correctness, RetrievalGroundedness
from mlflow.genai import scorer

# Custom RAG scorer using traces
@scorer
def context_utilization(trace, outputs: str) -> float:
    """Measure how much of the retrieved context was used"""
    from mlflow.entities import SpanType

    # Extract retriever spans
    retriever_spans = trace.search_spans(span_type=SpanType.RETRIEVER)
    if not retriever_spans:
        return 0.0

    # Get retrieved documents
    retrieved_docs = retriever_spans[0].outputs.get("documents", [])

    # Calculate utilization (simplified)
    utilized = sum(1 for doc in retrieved_docs if doc["content"] in outputs)
    return utilized / len(retrieved_docs) if retrieved_docs else 0.0

# RAG evaluation dataset
rag_dataset = [
    {
        "inputs": {"question": "What are MLflow's key features?"},
        "expectations": {
            "expected_response": "MLflow provides tracking, projects, models, and registry",
            "expected_sources": ["mlflow-docs-overview"]
        },
    },
]

# Run evaluation
results = mlflow.genai.evaluate(
    data=rag_dataset,
    predict_fn=rag_pipeline,
    scorers=[
        Correctness(),
        RetrievalGroundedness(),
        context_utilization,
    ],
)
```

### Relevance Evaluation

```python
from mlflow.metrics.genai import relevance

# Create relevance metric for RAG
relevance_metric = relevance(
    model="openai:/gpt-4",
    parameters={"temperature": 0.0}
)

# Configure evaluator
evaluator_config = {
    "col_mapping": {
        "inputs": "question",
        "context": "retrieved_context",
    }
}

results = mlflow.evaluate(
    model=rag_model,
    data=eval_data,
    extra_metrics=[relevance_metric],
    evaluator_config=evaluator_config,
)
```

### Chunk Size Optimization

```python
import mlflow

def evaluate_chunk_strategy(chunk_size: int):
    """Evaluate RAG performance with different chunk sizes"""

    with mlflow.start_run(run_name=f"chunk_size_{chunk_size}"):
        # Log chunk configuration
        mlflow.log_param("chunk_size", chunk_size)

        # Create retriever with chunk size
        retriever = create_retriever(chunk_size=chunk_size)
        rag_pipeline = create_rag_pipeline(retriever)

        # Evaluate
        results = mlflow.genai.evaluate(
            data=eval_dataset,
            predict_fn=rag_pipeline,
            scorers=[Correctness(), RetrievalGroundedness()],
        )

        # Log aggregate metrics
        mlflow.log_metrics({
            "mean_correctness": results.metrics["mean_correctness"],
            "mean_groundedness": results.metrics["mean_groundedness"],
        })

# Test different chunk sizes
for size in [500, 1000, 2000]:
    evaluate_chunk_strategy(size)
```

### Question Generation for Retrieval Evaluation

```python
import mlflow
from openai import OpenAI

def generate_evaluation_questions(documents: list) -> list:
    """Generate test questions from documents for retrieval evaluation"""

    client = OpenAI()
    questions = []

    for doc in documents:
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {
                    "role": "system",
                    "content": "Generate 3 questions that can be answered using this document."
                },
                {"role": "user", "content": doc["content"]}
            ]
        )

        doc_questions = response.choices[0].message.content.split("\n")
        for q in doc_questions:
            questions.append({
                "question": q,
                "source_doc_id": doc["id"],
                "ground_truth_chunks": [doc["id"]]
            })

    return questions
```

---

## 6. Best Practices

### LLM Experiment Tracking

1. **Structured Logging**
   ```python
   with mlflow.start_run():
       # Log all relevant parameters
       mlflow.log_params({
           "model": "gpt-4",
           "temperature": 0.7,
           "prompt_version": "v2.1",
           "retriever_k": 5,
       })

       # Log predictions for reproducibility
       mlflow.llm.log_predictions(inputs, outputs, prompts)
   ```

2. **Consistent Naming Conventions**
   - Use descriptive experiment names
   - Tag runs with environment, model version, purpose
   - Version prompts with semantic versioning

3. **Artifact Management**
   - Store prompt templates as artifacts
   - Log evaluation datasets
   - Archive model configurations

### Evaluation Workflows

1. **Evaluation-Driven Development**
   ```python
   # Define evaluation suite early
   eval_suite = [
       Correctness(),
       Guidelines(name="tone", guidelines="Professional tone"),
       custom_domain_scorer,
   ]

   # Run evaluation on every change
   results = mlflow.genai.evaluate(
       data=golden_dataset,
       predict_fn=model_v2,
       scorers=eval_suite,
   )

   # Compare with baseline
   if results.metrics["mean_correctness"] < baseline:
       raise ValueError("Regression detected")
   ```

2. **Combine Multiple Evaluation Types**
   - Use built-in scorers for common metrics
   - Create custom scorers for domain-specific needs
   - Leverage traces for deep inspection

3. **Maintain Golden Datasets**
   - Curate high-quality evaluation data
   - Include edge cases and failure modes
   - Update regularly with production examples

### Tracing Best Practices

1. **Use Appropriate Span Types**
   ```python
   @mlflow.trace(span_type=SpanType.RETRIEVER)
   def retrieve():
       pass

   @mlflow.trace(span_type=SpanType.LLM)
   def generate():
       pass
   ```

2. **Add Meaningful Attributes**
   ```python
   with mlflow.start_span(name="processing") as span:
       span.set_attribute("doc_count", len(docs))
       span.set_attribute("model_version", "v2.1")
   ```

3. **Production Optimization**
   - Enable async logging
   - Use lightweight tracing SDK
   - Implement sampling for high-volume apps

### Prompt Engineering

1. **Version Control Strategy**
   ```python
   # Use aliases for deployment stages
   mlflow.set_prompt_alias("qa-prompt", "development", version=5)
   mlflow.set_prompt_alias("qa-prompt", "staging", version=4)
   mlflow.set_prompt_alias("qa-prompt", "production", version=3)
   ```

2. **A/B Testing Prompts**
   ```python
   import random

   # Load different versions for testing
   version = "a" if random.random() < 0.5 else "b"
   prompt = mlflow.genai.load_prompt(f"prompts:/qa-prompt@test_{version}")

   # Log which version was used
   mlflow.log_param("prompt_version", version)
   ```

3. **Collaborative Development**
   - Use meaningful commit messages
   - Tag prompts with author and purpose
   - Review prompts before promoting to production

### RAG System Optimization

1. **Systematic Evaluation**
   - Evaluate retrieval and generation separately
   - Use precision@k and recall@k for retrieval
   - Use faithfulness for generation quality

2. **Iterate on Components**
   - Test different chunking strategies
   - Compare embedding models
   - Optimize retrieval parameters

3. **Monitor in Production**
   - Track context utilization
   - Monitor response groundedness
   - Alert on quality degradation

---

## API Reference Summary

### Core Modules

| Module | Purpose |
|--------|---------|
| `mlflow.llm` | LLM tracking utilities |
| `mlflow.genai` | Modern GenAI evaluation and management |
| `mlflow.genai.scorers` | Predefined evaluation scorers |
| `mlflow.genai.datasets` | Evaluation dataset management |
| `mlflow.genai.judges` | Custom LLM judge creation |
| `mlflow.metrics.genai` | GenAI metrics (legacy) |

### Key Functions

| Function | Purpose |
|----------|---------|
| `mlflow.llm.log_predictions()` | Log LLM inputs/outputs |
| `mlflow.genai.evaluate()` | Run GenAI evaluation |
| `mlflow.genai.register_prompt()` | Create/version prompts |
| `mlflow.genai.load_prompt()` | Load prompt by name/alias |
| `mlflow.trace()` | Decorator for manual tracing |
| `mlflow.start_span()` | Context manager for spans |
| `mlflow.log_expectation()` | Log ground truth |
| `mlflow.log_feedback()` | Log evaluation feedback |

### Version Requirements

- MLflow Tracing: >= 2.14.0
- Modern GenAI API: >= 3.0.0
- `make_judge` API: >= 3.4.0
- Evaluation Datasets: >= 3.2.0

---

## Resources

- [MLflow LLM Documentation](https://mlflow.org/docs/latest/llms/index.html)
- [GenAI Evaluation Guide](https://mlflow.org/docs/latest/genai/eval-monitor/)
- [Prompt Registry](https://mlflow.org/docs/latest/genai/prompt-registry/)
- [Tracing Documentation](https://mlflow.org/docs/latest/genai/tracing/)
- [RAG Tutorials](https://mlflow.org/docs/latest/llms/rag/index.html)
- [Python API Reference](https://mlflow.org/docs/latest/python_api/index.html)


---

# LLM Integrations

# MLflow LLM Framework Integrations

Comprehensive reference documentation for MLflow's integrations with LLM frameworks and tools. This covers tracing, autologging, model management, and deployment capabilities.

## Table of Contents

1. [Overview](#overview)
2. [LangChain Integration](#langchain-integration)
3. [OpenAI Integration](#openai-integration)
4. [Transformers/Hugging Face Integration](#transformershugging-face-integration)
5. [LlamaIndex Integration](#llamaindex-integration)
6. [Other LLM Integrations](#other-llm-integrations)
   - [Anthropic Claude](#anthropic-claude)
   - [Google Gemini](#google-gemini)
   - [AWS Bedrock](#aws-bedrock)
   - [Ollama (Local Models)](#ollama-local-models)
   - [Mistral AI](#mistral-ai)
   - [DSPy](#dspy)
7. [Multi-Agent Framework Integrations](#multi-agent-framework-integrations)
   - [AutoGen](#autogen)
   - [CrewAI](#crewai)
8. [MLflow AI Gateway / Deployments](#mlflow-ai-gateway--deployments)
9. [Vector Store & RAG Tracking](#vector-store--rag-tracking)
10. [Manual Tracing & Custom Instrumentation](#manual-tracing--custom-instrumentation)
11. [Common Patterns & Best Practices](#common-patterns--best-practices)

---

## Overview

MLflow provides comprehensive observability and tracking capabilities for LLM applications through:

- **Automatic Tracing**: One-line enablement via `mlflow.<library>.autolog()`
- **Token Usage Tracking**: Automatic capture of input/output tokens
- **Model Logging**: Persist and version LLM models and pipelines
- **Evaluation**: Built-in metrics for LLM quality assessment

### Version Requirements

Most tracing features require **MLflow 2.14.0+**, with enhanced token tracking available in **MLflow 3.1.0+**.

### General Autolog Pattern

```python
import mlflow

# Enable tracing for a specific library
mlflow.<library>.autolog()

# Set experiment for organization
mlflow.set_experiment("My LLM Experiment")

# Optionally set tracking URI
mlflow.set_tracking_uri("http://localhost:5000")
```

### Disabling Autolog

```python
# Disable for specific library
mlflow.<library>.autolog(disable=True)

# Disable all autologging
mlflow.autolog(disable=True)
```

---

## LangChain Integration

### Setup

```python
import mlflow

mlflow.langchain.autolog()
```

**Compatibility**: LangChain versions 0.3.7 through 0.3.27

### Autolog Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `disable` | bool | False | Toggle autologging on/off |
| `exclusive` | bool | False | Prevent logging to user-created fluent runs |
| `disable_for_unsupported_versions` | bool | False | Disable for untested LangChain versions |
| `silent` | bool | False | Suppress MLflow event logs and warnings |
| `log_traces` | bool | True | Capture execution traces via MlflowLangchainTracer |

### What Gets Traced

- `invoke`, `batch`, `stream`
- `ainvoke`, `abatch`, `astream`
- `get_relevant_documents` (retrievers)
- `__call__` (Chains and AgentExecutors)

### Token Usage Tracking

Available in MLflow 3.1.0+:

```python
# Individual LLM calls: mlflow.chat.tokenUsage span attribute
# Total usage: mlflow.trace.tokenUsage metadata field

trace = mlflow.get_trace(trace_id=last_trace_id)
total_usage = trace.info.token_usage
```

### LangGraph Support

MLflow automatically captures LangGraph graph executions when LangChain autolog is enabled:

```python
import mlflow
from langgraph.graph import StateGraph

mlflow.langchain.autolog()  # Also traces LangGraph

# Build and run your graph
graph = StateGraph(...)
app = graph.compile()
result = app.invoke({"input": "..."})
```

### Model Logging

```python
from langchain.chains import LLMChain
from langchain.llms import OpenAI

# Create chain
llm = OpenAI(temperature=0.9)
chain = LLMChain(llm=llm, prompt=prompt)

# Log to MLflow
mlflow.langchain.log_model(
    chain,
    "my_chain",
    input_example={"product": "colorful socks"},
    registered_model_name="my-langchain-model"
)

# Load model
loaded_chain = mlflow.langchain.load_model("runs:/<run_id>/my_chain")
```

### Custom Tracing with MlflowLangchainTracer

```python
from mlflow.langchain.langchain_tracer import MlflowLangchainTracer

class CustomTracer(MlflowLangchainTracer):
    def on_chat_model_start(self, serialized, messages, **kwargs):
        # Custom logic
        super().on_chat_model_start(serialized, messages, **kwargs)
```

---

## OpenAI Integration

### Setup

**Python:**
```python
import mlflow
from openai import OpenAI

mlflow.openai.autolog()

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

**TypeScript:**
```typescript
import { tracedOpenAI } from "mlflow-openai";
import OpenAI from "openai";

const client = tracedOpenAI(new OpenAI());
```

**Compatibility**: OpenAI SDK version 1.17+

### What Gets Captured

- Prompts and completion responses
- Latency measurements
- Model names and metadata (temperature, max_tokens)
- Function calling responses
- Built-in tools (web search, file search, computer use)
- Exceptions raised during execution

### Supported APIs

| API | Features Supported |
|-----|-------------------|
| Chat Completions | Normal, function calling, structured outputs, streaming (2.15.0+), async (2.21.0+) |
| Responses API | Web search, file search, computer use, reasoning (2.22.0+) |
| Embeddings | Normal and async |
| Agents SDK | Full support |

### Token Usage Tracking

Available in MLflow 3.1.0+:

```python
import mlflow
from openai import OpenAI

mlflow.openai.autolog()
client = OpenAI()

# For streaming, enable usage tracking
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    stream_options={"include_usage": True}  # Required for streaming
)

# Access token usage
trace = mlflow.get_trace(trace_id=mlflow.get_last_active_trace_id())
print(f"Total tokens: {trace.info.token_usage}")
```

### Tool/Function Calling

```python
from mlflow.entities import SpanType

@mlflow.trace(span_type=SpanType.TOOL)
def get_weather(location: str):
    """Get weather for a location."""
    return {"temperature": 72, "condition": "sunny"}

# Tool calls are automatically captured in traces
```

---

## Transformers/Hugging Face Integration

### Setup

```python
import mlflow
from transformers import pipeline

mlflow.transformers.autolog()
```

**Compatibility**: transformers versions 4.38.2 through 4.57.1

### Autolog Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `log_input_examples` | bool | False | Log input examples |
| `log_model_signatures` | bool | True | Log model signatures |
| `log_models` | bool | True | Log trained models |
| `log_datasets` | bool | True | Log dataset information |
| `disable` | bool | False | Disable autologging |

### Pipeline Logging

```python
from transformers import pipeline
import mlflow

# Create pipeline
text_generator = pipeline(
    "text-generation",
    model="gpt2",
    max_new_tokens=50
)

# Log to MLflow
with mlflow.start_run():
    mlflow.transformers.log_model(
        transformers_model=text_generator,
        artifact_path="text_generator",
        input_example=["Hello, I'm a language model"],
        signature=mlflow.models.infer_signature(
            ["Hello, I'm a language model"],
            ["Hello, I'm a language model that..."]
        )
    )
```

### Supported Pipeline Types

- Text generation and classification
- Summarization and translation
- Question answering and token classification
- Audio processing (ASR, audio classification)
- Feature extraction

**Note**: Computer vision and multi-modal models require native loading via `mlflow.transformers.load_model()`.

### Model Cards & Metadata

MLflow automatically:
- Fetches ModelCard from HuggingFace Hub
- Preserves license information
- Stores metadata about model types and components
- Infers signatures for supported pipeline types

### Storage-Efficient Logging

For unmodified pretrained models, use reference-only mode:

```python
mlflow.transformers.log_model(
    transformers_model=pipe,
    artifact_path="model",
    save_pretrained=False  # Only save reference, not weights
)
```

### PEFT/LoRA Support

MLflow 2.11.0+ supports Parameter-Efficient Fine-Tuning:

```python
from peft import LoraConfig, get_peft_model

# Only adapter weights are saved
mlflow.transformers.log_model(
    transformers_model=peft_model,
    artifact_path="lora_model"
)
```

### MLflowCallback for Training

```python
from transformers import Trainer, TrainingArguments
from mlflow.transformers import MLflowCallback

training_args = TrainingArguments(
    output_dir="./results",
    report_to=["mlflow"],  # Enable MLflow reporting
    # ... other args
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    callbacks=[MLflowCallback()]
)

trainer.train()
```

---

## LlamaIndex Integration

### Setup

```python
import mlflow

mlflow.llama_index.autolog()
mlflow.set_experiment("LlamaIndex")
```

### What Gets Traced

- Query engine invocations
- Chat engine interactions
- Retriever operations
- Workflow executions
- LLM calls within the pipeline

### Token Usage Tracking

Available in MLflow 3.2.0+:

```python
trace = mlflow.get_trace(trace_id=last_trace_id)
total_usage = trace.info.token_usage  # Aggregate across all LLM calls
```

### Supported Engine Types

| Engine Type | Load Method |
|-------------|-------------|
| Query Engine | `engine_type="query"` |
| Chat Engine | `engine_type="chat"` |
| Retriever | `engine_type="retriever"` |

### Model Logging

```python
from llama_index.core import VectorStoreIndex

# Create index and engine
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

# Log to MLflow
with mlflow.start_run():
    mlflow.llama_index.log_model(
        llama_index_model=index,
        artifact_path="index",
        engine_type="query",
        input_example="What is the capital of France?"
    )
```

### Settings Tracking

MLflow tracks the state of the LlamaIndex Settings object for reproducibility:

```python
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.chunk_size = 512

# These settings are captured when logging models
```

---

## Other LLM Integrations

### Anthropic Claude

```python
import anthropic
import mlflow

mlflow.anthropic.autolog()

client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello, Claude"}]
)
```

**Features:**
- Chat completions
- Function/tool calling (automatically captured in trace UI)
- Async operations (MLflow 2.21.0+)
- Token usage tracking (MLflow 3.2.0+)

**Limitations:** Streaming, images, and batch processing not yet supported.

### Google Gemini

```python
import google.generativeai as genai
import mlflow
import os

mlflow.gemini.autolog()

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
response = client.models.generate_content(
    model="gemini-1.5-flash",
    contents="The opposite of hot is"
)
```

**Features:**
- Text generation and chat
- Function calling
- Embeddings
- Async operations (MLflow 3.2.0+)
- Token usage tracking (MLflow 3.4.0+)

**SDK Support:** Both Google GenAI SDK and legacy Google AI Python SDK (migration recommended).

### AWS Bedrock

```python
import boto3
import mlflow

mlflow.bedrock.autolog()
mlflow.set_experiment("Bedrock")

bedrock = boto3.client(
    service_name="bedrock-runtime",
    region_name="us-east-1"
)

response = bedrock.converse(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{
        "role": "user",
        "content": "Describe 'hello world' in one line."
    }],
    inferenceConfig={
        "maxTokens": 512,
        "temperature": 0.1,
        "topP": 0.9
    }
)
```

**Supported APIs:**
- `converse` - Standard conversation
- `converse_stream` - Streaming conversation
- `invoke_model` - Direct model invocation
- `invoke_model_with_response_stream` - Streaming invocation

**Token Usage Tracking:** Automatic for Claude, Jamba, Titan/Nova, and Llama models.

**Streaming Note:** Spans are created when chunks are consumed, not when response is returned.

### Ollama (Local Models)

Ollama uses OpenAI-compatible endpoints, so use OpenAI autolog:

```python
import mlflow
from openai import OpenAI

mlflow.openai.autolog()
mlflow.set_experiment("Ollama")

# Point to Ollama's local endpoint
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="dummy"  # Ollama doesn't require API key
)

response = client.chat.completions.create(
    model="llama3.2:1b",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

**Token Usage:** Supported in MLflow 3.2.0+ for local endpoints.

**Note:** Native Ollama Python SDK requires custom tracing implementation.

### Mistral AI

```python
import mlflow
from mistralai.client import MistralClient

mlflow.mistral.autolog()

client = MistralClient()
response = client.chat(
    model="mistral-small-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

**Limitations:** Only synchronous Text Generation API. Async and streaming not traced.

### DSPy

```python
import dspy
import mlflow

mlflow.dspy.autolog()
mlflow.set_experiment("DSPy")

lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)

# Define and run module
class QA(dspy.Signature):
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

qa = dspy.Predict(QA)
result = qa(question="What is 2+2?")
```

**Autolog Parameters:**

| Parameter | Default | Description |
|-----------|---------|-------------|
| `log_traces` | True | Log traces for module invocations |
| `log_traces_from_eval` | True | Log during evaluation |
| `log_traces_from_compile` | False | Log during optimization (high volume) |
| `log_compiles` | False | Log optimization process info |

**Token Usage:** Supported in MLflow 3.5.0+

---

## Multi-Agent Framework Integrations

### AutoGen

```python
import mlflow
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

# Import classes BEFORE calling autolog
mlflow.autogen.autolog()
mlflow.set_experiment("AutoGen")

model_client = OpenAIChatCompletionClient(model="gpt-4.1-nano")

@mlflow.trace(span_type=SpanType.TOOL)
def add(a: int, b: int) -> int:
    return a + b

agent = AssistantAgent(
    name="assistant",
    model_client=model_client,
    system_message="You are a helpful assistant.",
    tools=[add]
)

await agent.run(task="What is 1+1?")
```

**What Gets Captured:**
- Which agent is called at different turns
- Messages passed between agents
- LLM and tool calls (organized per agent and turn)
- Latencies and exceptions

**Version Requirements:** AutoGen 0.4.9+ (use AG2 integration for AutoGen 0.2)

**Token Usage:** Available in MLflow 3.2.0+

**Limitations:** Async streaming APIs (`run_stream`, `on_messages_stream`) not traced.

### CrewAI

```python
import mlflow
from crewai import Agent, Task, Crew

mlflow.crewai.autolog()
mlflow.set_experiment("CrewAI")

researcher = Agent(
    role='Researcher',
    goal='Research and analyze data',
    backstory='Expert data researcher'
)

task = Task(
    description='Research AI trends',
    agent=researcher
)

crew = Crew(
    agents=[researcher],
    tasks=[task]
)

result = crew.kickoff()
```

**What Gets Captured:**
- Tasks and executing agents
- LLM calls with prompts, responses, and metadata
- Memory load/write operations
- Latencies and exceptions

**Features:**
- OpenTelemetry compatibility (export to Jaeger, Zipkin, AWS X-Ray)
- Model packaging and deployment
- Built-in evaluation via `mlflow.evaluate()`

**Limitations:** Only synchronous task execution. Async kickoff not supported.

---

## MLflow AI Gateway / Deployments

> **Note:** MLflow AI Gateway has been deprecated. Use MLflow Deployments for LLMs instead.

### Overview

The MLflow Deployments Server provides:
- Unified interface for multiple LLM providers
- Centralized API key management
- Rate limiting and access control
- Hot-swapping endpoints without downtime

### Configuration File (YAML)

```yaml
endpoints:
  - name: chat
    endpoint_type: llm/v1/chat
    model:
      provider: openai
      name: gpt-4
      config:
        openai_api_key: $OPENAI_API_KEY
    limit:
      renewal_period: minute
      calls: 10

  - name: embeddings
    endpoint_type: llm/v1/embeddings
    model:
      provider: openai
      name: text-embedding-ada-002
      config:
        openai_api_key: $OPENAI_API_KEY

  - name: claude
    endpoint_type: llm/v1/chat
    model:
      provider: anthropic
      name: claude-sonnet-4-20250514
      config:
        anthropic_api_key: $ANTHROPIC_API_KEY
```

### Supported Providers

- OpenAI (including Azure OpenAI with AAD)
- Anthropic
- Cohere
- MosaicML
- Hugging Face Text Generation Inference
- AI21 Labs
- Amazon Bedrock
- Mistral

### Rate Limiting

```yaml
limit:
  renewal_period: minute  # second|minute|hour|day|month|year
  calls: 10
```

Returns HTTP 429 when limit exceeded.

### API Key Management Options

1. **Direct in YAML** (not recommended for production)
2. **Environment variable**: `$OPENAI_API_KEY`
3. **File reference**: Path to key file

### Security Recommendations

- Restrict server access
- Store config files securely
- Use environment variables for API keys
- Implement proper access controls

### Starting the Server

```bash
mlflow deployments start-server --config-path config.yaml
```

### Querying Endpoints

**Python Client:**
```python
from mlflow.deployments import get_deploy_client

client = get_deploy_client("http://localhost:5000")

response = client.predict(
    endpoint="chat",
    inputs={
        "messages": [{"role": "user", "content": "Hello!"}]
    }
)
```

**REST API:**
```bash
curl -X POST http://localhost:5000/endpoints/chat/invocations \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}]}'
```

---

## Vector Store & RAG Tracking

### RAG System Tracking with MLflow

MLflow provides tracking for RAG applications through integration with vector stores like ChromaDB.

### Setting Up RAG Tracking

```python
import mlflow
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA

mlflow.langchain.autolog()

# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
    documents,
    embeddings,
    collection_name="docs"
)

# Create retrieval chain
retriever = vectorstore.as_retriever()
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever
)

# Queries are automatically traced
result = qa_chain.invoke({"query": "What is MLflow?"})
```

### Versioning Vector Stores

Track changes to your document corpus:

```python
with mlflow.start_run():
    # Log vector store configuration
    mlflow.log_params({
        "chunk_size": 1000,
        "embedding_model": "text-embedding-ada-002",
        "num_documents": len(documents)
    })

    # Log the index as an artifact
    vectorstore.persist()
    mlflow.log_artifacts("./chroma_db", "vector_store")
```

### Evaluating RAG Systems

```python
import mlflow

# Define evaluation data
eval_data = [
    {"question": "What is MLflow?", "ground_truth": "..."},
    # ...
]

# Evaluate with built-in metrics
results = mlflow.evaluate(
    model=qa_chain,
    data=eval_data,
    targets="ground_truth",
    model_type="question-answering",
    evaluators=["default"],
    extra_metrics=[
        mlflow.metrics.latency(),
        mlflow.metrics.token_count()
    ]
)
```

### Best Practices for RAG Tracking

1. **Version your index**: Track document additions/removals
2. **Log embedding parameters**: chunk_size, overlap, model
3. **Track retrieval metrics**: relevance scores, latency
4. **Evaluate chunking strategies**: Compare different configurations
5. **Monitor token usage**: Track costs across the pipeline

---

## Manual Tracing & Custom Instrumentation

### Using Decorators

```python
import mlflow
from mlflow.entities import SpanType

@mlflow.trace(
    name="my_function",
    span_type=SpanType.TOOL,
    attributes={"key": "value"}
)
def my_tool(x: int, y: int) -> int:
    return x + y
```

### Available Span Types

- `SpanType.LLM` - Language model calls
- `SpanType.CHAIN` - Chain/workflow steps
- `SpanType.TOOL` - Tool/function calls
- `SpanType.AGENT` - Agent operations
- `SpanType.RETRIEVER` - Retrieval operations
- `SpanType.EMBEDDING` - Embedding generation
- `SpanType.PARSER` - Output parsing

### Dynamic Attribute Setting

```python
@mlflow.trace(span_type=SpanType.LLM)
def invoke(prompt: str):
    model_id = "gpt-4o-mini"

    # Get current span and set attributes
    span = mlflow.get_current_active_span()
    span.set_attributes({
        "model": model_id,
        "temperature": 0.7
    })

    return client.invoke(prompt, model=model_id)
```

### Context Manager for Code Blocks

```python
with mlflow.start_span(name="my_operation") as span:
    span.set_inputs({"x": x, "y": y})

    # Your code here
    result = complex_operation(x, y)

    span.set_outputs(result)
    span.set_attribute("intermediate_value", intermediate)
```

### Combining Auto and Manual Tracing

```python
import mlflow
from openai import OpenAI

mlflow.openai.autolog()

@mlflow.trace(name="my_app", span_type=SpanType.CHAIN)
def my_application(user_input: str):
    # OpenAI calls are auto-traced as child spans
    client = OpenAI()

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": user_input}]
    )

    return response.choices[0].message.content
```

### Low-Level Client API

For advanced control:

```python
from mlflow import MlflowClient

client = MlflowClient()

# Start trace
trace = client.start_trace(
    name="my_trace",
    inputs={"input_key": "input_value"}
)

# Start child span
span = client.start_span(
    name="child_span",
    request_id=trace.info.request_id,
    parent_id=trace.info.root_span_id,
    inputs={"span_input": "value"}
)

# End span
client.end_span(
    request_id=span.request_id,
    span_id=span.span_id,
    outputs={"output_key": "output_value"},
    attributes={"custom_attr": "value"}
)

# End trace
client.end_trace(
    request_id=trace.info.request_id,
    outputs={"final_output": "value"}
)
```

---

## Common Patterns & Best Practices

### 1. Organizing Experiments

```python
import mlflow

# Set experiment by name
mlflow.set_experiment("Production/LLM-Chatbot")

# Or by ID
mlflow.set_experiment(experiment_id="123")

# Set tags for filtering
mlflow.set_tags({
    "environment": "production",
    "model_type": "chat",
    "version": "1.0.0"
})
```

### 2. Accessing Traces Programmatically

```python
# Get last trace
last_trace_id = mlflow.get_last_active_trace_id()
trace = mlflow.get_trace(trace_id=last_trace_id)

# Access spans
for span in trace.data.spans:
    print(f"Span: {span.name}")
    print(f"  Duration: {span.end_time_ns - span.start_time_ns}ns")
    print(f"  Inputs: {span.inputs}")
    print(f"  Outputs: {span.outputs}")
```

### 3. Error Handling

Traces automatically capture exceptions:

```python
@mlflow.trace()
def risky_operation():
    try:
        # Your code
        result = call_api()
    except Exception as e:
        # Exception is automatically logged to span
        raise

# View exceptions in MLflow UI
```

### 4. Production Deployment Pattern

```python
import mlflow

# Disable autolog in production if needed
mlflow.openai.autolog(disable=True)

# Use manual tracing for specific operations
@mlflow.trace(name="production_inference")
def inference(request):
    # Production code
    pass
```

### 5. Multi-Model Comparison

```python
import mlflow

models = ["gpt-4o-mini", "claude-sonnet-4-20250514", "gemini-1.5-flash"]

for model in models:
    with mlflow.start_run(run_name=f"eval_{model}"):
        mlflow.log_param("model", model)

        # Run evaluation
        metrics = evaluate_model(model)

        mlflow.log_metrics(metrics)
```

### 6. Cost Tracking

```python
# Token usage is automatically captured
trace = mlflow.get_trace(trace_id=trace_id)
token_usage = trace.info.token_usage

# Calculate cost (example pricing)
COST_PER_1K_INPUT = 0.001
COST_PER_1K_OUTPUT = 0.003

cost = (
    token_usage.input_tokens * COST_PER_1K_INPUT / 1000 +
    token_usage.output_tokens * COST_PER_1K_OUTPUT / 1000
)

mlflow.log_metric("estimated_cost", cost)
```

### 7. OpenTelemetry Export

Export traces to external systems:

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Configure exporter
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# MLflow traces will be exported
```

---

## Quick Reference: All Autolog Functions

| Library | Autolog Function | Min MLflow Version | Token Tracking |
|---------|-----------------|-------------------|----------------|
| LangChain | `mlflow.langchain.autolog()` | 2.14.0 | 3.1.0 |
| OpenAI | `mlflow.openai.autolog()` | 2.14.0 | 3.1.0 |
| Transformers | `mlflow.transformers.autolog()` | 2.14.0 | N/A |
| LlamaIndex | `mlflow.llama_index.autolog()` | 2.14.0 | 3.2.0 |
| Anthropic | `mlflow.anthropic.autolog()` | 2.17.0 | 3.2.0 |
| Gemini | `mlflow.gemini.autolog()` | 2.17.0 | 3.4.0 |
| Bedrock | `mlflow.bedrock.autolog()` | 2.17.0 | 3.1.0 |
| Mistral | `mlflow.mistral.autolog()` | 2.17.0 | N/A |
| DSPy | `mlflow.dspy.autolog()` | 2.17.0 | 3.5.0 |
| AutoGen | `mlflow.autogen.autolog()` | 3.1.0 | 3.2.0 |
| CrewAI | `mlflow.crewai.autolog()` | 3.0.0 | N/A |

---

## Resources

### Official Documentation
- [MLflow Tracing Overview](https://mlflow.org/docs/latest/llms/tracing/index.html)
- [MLflow GenAI Integrations](https://mlflow.org/docs/latest/genai/tracing/integrations/)
- [MLflow LLM Evaluation](https://mlflow.org/docs/latest/llms/llm-evaluate/)

### API References
- [mlflow.langchain](https://mlflow.org/docs/latest/python_api/mlflow.langchain.html)
- [mlflow.openai](https://mlflow.org/docs/latest/python_api/mlflow.openai.html)
- [mlflow.transformers](https://mlflow.org/docs/latest/python_api/mlflow.transformers.html)
- [mlflow.llama_index](https://mlflow.org/docs/latest/python_api/mlflow.llama_index.html)

### Tutorials & Guides
- [RAG Evaluation with MLflow](https://mlflow.org/docs/latest/llms/rag/notebooks/mlflow-e2e-evaluation/)
- [Building Advanced RAG with LlamaIndex](https://mlflow.org/blog/mlflow-llama-index-workflow)
- [Custom Tracing Guide](https://mlflow.org/blog/custom-tracing)

---

*Last updated: November 2025*
*MLflow version coverage: 2.14.0 - 3.5.0*
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall