YemenJPT/YemenJPT-AI-Skills / skills/core/azure-ai-voicelive-py
azure-ai-voicelive-py
VerifiedBuild real-time voice AI applications using Azure AI Voice Live SDK (azure-ai-voicelive). Use this skill when creating Python applications that need real-time bidirectional audio communication with Azure AI, including voice assistants, voice-enabled chatbots, real-time speech-to-speech translation, voice-driven avatars, or any WebSocket-based audio streaming with AI models. Supports Server VAD (Voice Activity Detection), turn-based conversation, function calling, MCP tools, avatar integration, and transcription.
| File | Kind | Size |
|---|---|---|
| README.ar.md | markdown | 4.5 KB |
| README.en.md | markdown | 3.2 KB |
| SKILL.md | skill md | 13.7 KB |
| references/api-reference.md | markdown | 8.7 KB |
| references/examples.md | markdown | 16.9 KB |
| references/models.md | markdown | 18.0 KB |
references/examples.md
View on GitHub# Azure AI Voice Live SDK - Examples
## Table of Contents
- [Basic Voice Assistant](#basic-voice-assistant)
- [Function Calling](#function-calling)
- [Manual Turn Control](#manual-turn-control)
- [Audio File Processing](#audio-file-processing)
- [Interrupt Handling](#interrupt-handling)
- [Multi-modal (Text + Audio)](#multi-modal-text--audio)
- [Azure Voice Integration](#azure-voice-integration)
- [Avatar Integration](#avatar-integration)
- [Transcription Only](#transcription-only)
---
## Basic Voice Assistant
Complete voice assistant with Server VAD.
```python
import asyncio
import base64
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def voice_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
# Configure session
await conn.session.update(session={
"instructions": "You are a helpful voice assistant. Be concise.",
"modalities": ["text", "audio"],
"voice": "alloy",
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"silence_duration_ms": 500
},
"input_audio_transcription": {
"model": "whisper-1"
}
})
# Start microphone input (pseudo-code)
mic_task = asyncio.create_task(stream_microphone(conn))
# Process events
async for event in conn:
match event.type:
case "session.created":
print("Session ready")
case "input_audio_buffer.speech_started":
print("π€ Listening...")
case "conversation.item.input_audio_transcription.completed":
print(f"You: {event.transcript}")
case "response.audio.delta":
audio = base64.b64decode(event.delta)
await play_audio(audio)
case "response.audio_transcript.done":
print(f"Assistant: {event.transcript}")
case "error":
print(f"Error: {event.error.message}")
break
async def stream_microphone(conn):
"""Stream microphone audio to the connection."""
async for chunk in read_microphone(): # Your audio capture
b64 = base64.b64encode(chunk).decode()
await conn.input_audio_buffer.append(audio=b64)
asyncio.run(voice_assistant())
```
---
## Function Calling
Voice assistant with tool use.
```python
import asyncio
import json
import base64
from azure.ai.voicelive.aio import connect
from azure.ai.voicelive.models import FunctionTool
from azure.core.credentials import AzureKeyCredential
# Define tools
TOOLS = [
FunctionTool(
type="function",
name="get_weather",
description="Get current weather for a location",
parameters={
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and state, e.g. 'San Francisco, CA'"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"default": "fahrenheit"
}
},
"required": ["location"]
}
),
FunctionTool(
type="function",
name="set_reminder",
description="Set a reminder for the user",
parameters={
"type": "object",
"properties": {
"message": {"type": "string"},
"time": {"type": "string", "description": "ISO 8601 datetime"}
},
"required": ["message", "time"]
}
)
]
def handle_function_call(name: str, arguments: str) -> dict:
"""Execute function and return result."""
args = json.loads(arguments)
if name == "get_weather":
# Mock weather API
return {
"location": args["location"],
"temperature": 72,
"unit": args.get("unit", "fahrenheit"),
"conditions": "sunny"
}
elif name == "set_reminder":
# Mock reminder service
return {"status": "success", "reminder_id": "123"}
else:
return {"error": f"Unknown function: {name}"}
async def function_calling_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"instructions": "You can check weather and set reminders.",
"modalities": ["text", "audio"],
"voice": "alloy",
"tools": TOOLS,
"tool_choice": "auto"
})
async for event in conn:
match event.type:
case "response.function_call_arguments.done":
# Execute the function
result = handle_function_call(event.name, event.arguments)
# Send result back
await conn.conversation.item.create(item={
"type": "function_call_output",
"call_id": event.call_id,
"output": json.dumps(result)
})
# Continue the conversation
await conn.response.create()
case "response.audio.delta":
audio = base64.b64decode(event.delta)
await play_audio(audio)
case "response.done":
if event.response.status == "completed":
print("Response complete")
asyncio.run(function_calling_assistant())
```
---
## Manual Turn Control
Push-to-talk style without VAD.
```python
import asyncio
import base64
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def push_to_talk():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
# Disable VAD for manual control
await conn.session.update(session={
"instructions": "You are a helpful assistant.",
"modalities": ["text", "audio"],
"voice": "alloy",
"turn_detection": None # Disable VAD
})
# Simulate push-to-talk
while True:
input("Press Enter to start recording...")
# Record audio (simulate with chunks)
chunks = await record_audio_until_release()
# Send all audio
for chunk in chunks:
b64 = base64.b64encode(chunk).decode()
await conn.input_audio_buffer.append(audio=b64)
# Commit and request response
await conn.input_audio_buffer.commit()
await conn.response.create()
# Wait for response
async for event in conn:
if event.type == "response.audio.delta":
audio = base64.b64decode(event.delta)
await play_audio(audio)
elif event.type == "response.done":
break
asyncio.run(push_to_talk())
```
---
## Audio File Processing
Process an audio file and get a response.
```python
import asyncio
import base64
from pathlib import Path
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def process_audio_file(audio_path: str):
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"instructions": "Respond to the audio message.",
"modalities": ["text", "audio"],
"voice": "alloy",
"turn_detection": None,
"input_audio_transcription": {"model": "whisper-1"}
})
# Read and send audio file
audio_data = Path(audio_path).read_bytes()
# Send in chunks (24kHz * 2 bytes * 0.1s = 4800 bytes)
chunk_size = 4800
for i in range(0, len(audio_data), chunk_size):
chunk = audio_data[i:i + chunk_size]
b64 = base64.b64encode(chunk).decode()
await conn.input_audio_buffer.append(audio=b64)
# Commit and request response
await conn.input_audio_buffer.commit()
await conn.response.create()
# Collect response
response_audio = bytearray()
response_text = ""
user_transcript = ""
async for event in conn:
match event.type:
case "conversation.item.input_audio_transcription.completed":
user_transcript = event.transcript
case "response.audio.delta":
response_audio.extend(base64.b64decode(event.delta))
case "response.audio_transcript.done":
response_text = event.transcript
case "response.done":
break
return {
"user_said": user_transcript,
"assistant_said": response_text,
"audio": bytes(response_audio)
}
result = asyncio.run(process_audio_file("input.pcm"))
print(f"User: {result['user_said']}")
print(f"Assistant: {result['assistant_said']}")
Path("output.pcm").write_bytes(result['audio'])
```
---
## Interrupt Handling
Handle user interruptions gracefully.
```python
import asyncio
import base64
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def interruptible_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"instructions": "You are a helpful assistant.",
"modalities": ["text", "audio"],
"voice": "alloy",
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"silence_duration_ms": 500
}
})
is_responding = False
async for event in conn:
match event.type:
case "response.created":
is_responding = True
case "response.done":
is_responding = False
case "input_audio_buffer.speech_started":
if is_responding:
# User interrupted - stop current response
print("π Interrupt detected!")
await conn.response.cancel()
await conn.output_audio_buffer.clear()
case "response.audio.delta":
if is_responding:
audio = base64.b64decode(event.delta)
await play_audio(audio)
asyncio.run(interruptible_assistant())
```
---
## Multi-modal (Text + Audio)
Send text context, receive audio response.
```python
import asyncio
import base64
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def multimodal_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"instructions": "You are a helpful assistant.",
"modalities": ["text", "audio"],
"voice": "alloy",
"turn_detection": None
})
# Add context via text
await conn.conversation.item.create(item={
"type": "message",
"role": "system",
"content": [{"type": "input_text", "text": "The user's name is Alice."}]
})
# Add user message as text
await conn.conversation.item.create(item={
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What's my name?"}]
})
# Request audio response
await conn.response.create()
async for event in conn:
if event.type == "response.audio.delta":
audio = base64.b64decode(event.delta)
await play_audio(audio)
elif event.type == "response.done":
break
asyncio.run(multimodal_assistant())
```
---
## Azure Voice Integration
Use Azure Text-to-Speech voices.
```python
import asyncio
from azure.ai.voicelive.aio import connect
from azure.ai.voicelive.models import AzureStandardVoice, AzureCustomVoice
from azure.core.credentials import AzureKeyCredential
async def azure_voice_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
# Use Azure standard voice
await conn.session.update(session={
"instructions": "You are a helpful assistant.",
"modalities": ["text", "audio"],
"voice": AzureStandardVoice(
type="azure-standard",
name="en-US-JennyNeural"
)
})
# Or use custom voice
# await conn.session.update(session={
# "voice": AzureCustomVoice(
# type="azure-custom",
# endpoint_id="your-custom-voice-endpoint",
# name="YourCustomVoice"
# )
# })
async for event in conn:
# ... handle events
pass
asyncio.run(azure_voice_assistant())
```
---
## Avatar Integration
Connect to Azure Avatar for visual output.
```python
import asyncio
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def avatar_assistant():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"instructions": "You are a helpful assistant.",
"modalities": ["text", "audio", "avatar"],
"voice": "alloy",
"avatar": {
"type": "video-avatar",
"character": "lisa",
"output_protocol": "webrtc"
}
})
# Connect avatar
await conn.send({
"type": "session.avatar.connect"
})
async for event in conn:
match event.type:
case "session.avatar.connecting":
ice_servers = event.ice_servers
# Use ice_servers for WebRTC connection
print(f"Avatar connecting with {len(ice_servers)} ICE servers")
case "response.audio.delta":
# Audio is streamed via WebRTC, not this event
pass
asyncio.run(avatar_assistant())
```
---
## Transcription Only
Speech-to-text without AI response.
```python
import asyncio
import base64
from azure.ai.voicelive.aio import connect
from azure.core.credentials import AzureKeyCredential
async def transcription_only():
async with connect(
endpoint="https://eastus.api.cognitive.microsoft.com",
credential=AzureKeyCredential("YOUR_KEY"),
model="gpt-4o-realtime-preview"
) as conn:
await conn.session.update(session={
"modalities": ["text"], # No audio output
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"silence_duration_ms": 1000
},
"input_audio_transcription": {
"model": "whisper-1"
}
})
# Stream microphone
mic_task = asyncio.create_task(stream_microphone(conn))
transcripts = []
async for event in conn:
match event.type:
case "conversation.item.input_audio_transcription.delta":
print(event.delta, end="", flush=True)
case "conversation.item.input_audio_transcription.completed":
print() # Newline
transcripts.append(event.transcript)
return transcripts
asyncio.run(transcription_only())
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path β even casually (like "the xlsx in my downloads") β and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |