In the last article, we connected Misty to Claude and taught it reinforced prompts. Now we go one step further: we connect Misty to Azure AI Foundry — Microsoft's model catalog with over 1,700 models. The goal: Misty should not be locked to a single model, but choose the best model for each task. See with GPT-5.2 Vision, hear with Whisper v3, think with GPT-5.2, speak with OpenAI TTS. All orchestrated via Python.
What is Azure AI Foundry?
Azure AI Foundry (formerly Azure AI Studio) is Microsoft's central platform for AI models. Instead of configuring each API separately, Foundry gives you unified access to:
- OpenAI models — GPT-5.2, o4-mini, o3, DALL-E 4, Whisper v3, TTS
- Open-source models — Llama 3.3, Mistral Large, Phi-4, DeepSeek-R1, Stable Diffusion
- Microsoft models — Phi-4 (Small Language Model), Azure Speech
- Serverless API endpoints — Use models without your own infrastructure
- Managed Compute — Deploy your own models on GPU VMs
For Misty, this means: we can choose the optimal model for each sensory modality — and swap it when needed without touching the rest of the architecture.
The Multimodal Architecture
Azure AI Foundry
+-----------------------------------+
| |
+-------------+ | +-----------+ +--------------+ |
| Misty II |---->| | Whisper v3| | GPT-5.2 | |
| | | | (Hear) | | (See) | |
| Camera | | +-----+-----+ +------+-------+ |
| Microphone | | | | |
| Sensors | | +-----v---------------v-------+ |
| |<----| | GPT-5.2 | |
| REST API | | | (Multimodal Reasoning) | |
| | | +-------------+---------------+ |
+-------------+ | | |
^ | +------------v----------------+ |
| | | OpenAI TTS / Whisper v3 | |
+-------------| | (Speak) | |
| +-----------------------------+ |
+-----------------------------------+
^
+---------+---------+
| Python Orchestr. |
| + Model Router |
+-------------------+The Model Router: Right Model for Every Task
The core of our architecture: a Model Router that automatically selects the right model per sensory modality.
import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import (
SystemMessage, UserMessage, ImageContentItem,
ImageUrl, TextContentItem
)
from azure.core.credentials import AzureKeyCredential
class ModelRouter:
"""Automatically selects the best OpenAI model for each task – with client caching."""
def __init__(self):
self.foundry_endpoint = os.environ["AZURE_AI_FOUNDRY_ENDPOINT"]
self.api_key = os.environ["AZURE_AI_FOUNDRY_KEY"]
# Client cache for better performance
self.clients: dict = {}
self.models = {
"reasoning": "gpt-5.2",
"reasoning_fast": "o4-mini",
"vision": "gpt-5.2",
"vision_detailed": "gpt-5.2",
"speech_to_text": "whisper-v3",
"text_to_speech": "tts-1-hd",
"embedding": "text-embedding-3-large",
}
def get_client(self, model_name: str) -> ChatCompletionsClient:
if model_name not in self.clients:
self.clients[model_name] = ChatCompletionsClient(
endpoint=self.foundry_endpoint,
credential=AzureKeyCredential(self.api_key),
)
return self.clients[model_name]
def reason(self, system_prompt: str, user_input: str,
fast: bool = False) -> str:
model = self.models["reasoning_fast" if fast else "reasoning"]
client = self.get_client(model)
response = client.complete(
model=model,
messages=[
SystemMessage(content=system_prompt),
UserMessage(content=user_input),
],
temperature=0.3,
max_tokens=1024,
)
return response.choices[0].message.content
def see(self, image_bytes: bytes, question: str,
detailed: bool = False) -> str:
model = self.models["vision_detailed" if detailed else "vision"]
client = self.get_client(model)
import base64
img_b64 = base64.b64encode(image_bytes).decode()
response = client.complete(
model=model,
messages=[
UserMessage(content=[
TextContentItem(text=question),
ImageContentItem(
image_url=ImageUrl(
url=f"data:image/jpeg;base64,{img_b64}"
)
),
]),
],
max_tokens=512,
)
return response.choices[0].message.content
def hear(self, audio_bytes: bytes) -> str:
"""Whisper v3: Audio to text – safe temp file handling"""
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=self.foundry_endpoint,
api_key=self.api_key,
api_version="2025-01-01",
)
import tempfile
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
f.write(audio_bytes)
temp_path = f.name
try:
with open(temp_path, "rb") as audio_file:
transcript = client.audio.transcriptions.create(
model="whisper-v3",
file=audio_file,
language="en",
)
return transcript.text
finally:
if os.path.exists(temp_path):
os.unlink(temp_path)GPT-5.2 as Multimodal Orchestrator
The key insight: GPT-5.2 can process text, images, and structured data in a single request. We send the camera capture and the transcript together and get a holistic decision:
MULTIMODAL_SYSTEM_PROMPT = """You are Misty, a robot at CNEXT.
You receive simultaneously: a camera image, a speech transcript,
and sensor data. Make ONE decision based on ALL inputs.
Always respond as JSON:
{
"situation_summary": "What is happening (1-2 sentences)",
"primary_input": "vision|speech|sensor",
"action": {
"type": "speak|move|express|navigate|wait|multimodal",
"speech": "What you say",
"emotion": "happy|curious|thinking|surprised|neutral",
"movement": {"head_pitch": 0, "head_yaw": 0, "drive": 0},
"reason": "Why this action"
},
"confidence": 0.0-1.0,
"follow_up": "What you want to observe next"
}
SAFETY RULES:
- Never faster than drive=30
- At confidence < 0.4: action.type = "wait"
- On conflicting inputs (image shows X, speech says Y):
Prioritize speech, but mention the contradiction
"""GPT-5.2 Vision: Specialized Computer Vision
GPT-5.2 is not just good for general image descriptions — it also handles precise tasks like object counting, OCR, and region detection. Through specialized prompts, we get structured results from the same model:
class DetailedVision:
"""GPT-5.2 for specialized vision tasks with structured prompts."""
def __init__(self, router: ModelRouter):
self.router = router
def _parse_detections(self, result: str) -> list:
"""Parse the structured JSON response."""
try:
import json
return json.loads(result) if isinstance(result, str) else result
except Exception:
return []
def detect_objects(self, image_bytes: bytes) -> list:
result = self.router.see(
image_bytes,
"Detect all objects in the image. Respond as a JSON array "
"with {label, confidence, bbox: [x1,y1,x2,y2]} per object.",
detailed=True,
)
return self._parse_detections(result)
def read_text(self, image_bytes: bytes) -> str:
return self.router.see(
image_bytes,
"Read all visible text in the image. "
"Return only the detected text, line by line.",
detailed=True,
)
def describe_region(self, image_bytes: bytes,
bbox: tuple) -> str:
x1, y1, x2, y2 = bbox
return self.router.see(
image_bytes,
f"Describe the image region at coordinates "
f"[{x1},{y1},{x2},{y2}] in detail.",
detailed=True,
)
def count_people(self, image_bytes: bytes) -> int:
detections = self.detect_objects(image_bytes)
return sum(
1 for d in detections
if d.get("label") in ["person", "face"]
)
def detect_gestures(self, image_bytes: bytes) -> list:
result = self.router.see(
image_bytes,
"What gestures are the people in the image showing? "
"Waving, pointing, thumbs up, nodding?",
)
return result.split(", ") if isinstance(result, str) else [result]Why GPT-5.2 for everything? Instead of deploying separate vision models, we use GPT-5.2 with specialized prompts for each task. One model, one endpoint, maximum flexibility. For object detection and OCR, GPT-5.2 delivers comparable results to specialized models — with significantly simpler setup.
Adaptive Model Swapping at Runtime
The most powerful feature of our architecture: models can be swapped at runtime — no restart, no code change:
class AdaptiveModelRouter(ModelRouter):
"""Selects models dynamically based on context."""
def __init__(self):
super().__init__()
self.performance_log = []
def select_model(self, task: str, context: dict) -> str:
battery = context.get("battery", 100)
latency_budget = context.get("max_latency_ms", 2000)
crowd_size = context.get("people_count", 0)
if task == "reasoning":
if battery < 20 or latency_budget < 500:
return "o4-mini"
if crowd_size > 3:
return "gpt-5.2"
return "o4-mini"
if task == "vision":
if context.get("need_ocr") or context.get("need_counting"):
return "gpt-5.2"
if context.get("need_emotion"):
return "gpt-5.2"
return "o4-mini"
if task == "speech_to_text":
return "whisper-v3"
return self.models.get(task, "o4-mini")
def log_performance(self, model: str, task: str,
latency_ms: float, success: bool):
self.performance_log.append({
"model": model, "task": task,
"latency_ms": latency_ms, "success": success,
})
if len(self.performance_log) > 100:
self._optimize_routing()Results: Model Routing in Practice
| Task | Model | Latency | Cost/1K Requests |
|---|---|---|---|
| Quick response (small talk) | o4-mini | ~120ms | ~CHF 0.05 |
| Complex decision | GPT-5.2 | ~350ms | ~CHF 0.90 |
| Object detection (Vision) | GPT-5.2 | ~300ms | ~CHF 0.80 |
| Speech recognition | Whisper v3 | ~150ms | ~CHF 0.08 |
| Multimodal analysis | GPT-5.2 (image+text) | ~400ms | ~CHF 1.80 |
| Speech output | OpenAI TTS-1-HD | ~130ms | ~CHF 0.06 |
Through adaptive routing, we save about 60% of API costs compared to a pure GPT-5.2 solution, because simple tasks go to cheaper, faster models.
Why Azure AI Foundry Instead of Individual APIs?
| Aspect | Individual APIs | Azure AI Foundry |
|---|---|---|
| Authentication | One API key per provider | One endpoint, one key |
| Swap models | New SDK, new code | Just change model ID |
| Data residency | Unclear, check per provider | Switzerland North / West |
| Monitoring | Build yourself | Azure Monitor + AI Metrics |
| Compliance | Each contract separate | One Azure contract, GDPR-compliant |
Interested in Azure AI Foundry, multimodal robotics, or a Misty demo?
Further Reading
Internal:
External: