I'm Luis Castillo, Solution Engineer at CNEXT. But before I was talking about security and Zero Trust, I taught a robot to think. This is the story of how, as a student, I transformed the Misty II robot from a programmable toy into an autonomous AI system – one that sees, hears, speaks and makes independent decisions.

How It All Started: A Student and a Robot

When I started my studies, "Embodied AI" was just a buzzword to me. Robots that can think? Sounded like science fiction. Then CNEXT put a Misty II robot on my desk and said: "Make something out of it."

Misty II is a small, cute robot with big eyes, an LED display for a face, cameras, microphones, touch sensors and the ability to move autonomously. But out of the box, Misty can't do much more than play pre-programmed animations and react to simple commands. The brain was missing.

My goal was clear: I wanted to make Misty a robot that sees, hears, understands, speaks and acts autonomously. Not with pre-programmed answers – but with real AI.

The Architecture: OpenAI as Misty's Brain

The key insight was: Misty doesn't need a brain on local hardware – the cloud is the brain. I developed an architecture that connects Misty's sensors with OpenAI and Azure AI Services:

System Overview

The Data Flow

1. 1Misty hears: The 3 microphones capture audio
2. 2Audio → Text: Azure Speech-to-Text transcribes in real-time
3. 3Text → GPT-5 / Claude Opus: The text is sent along with context (recognized people, current position, conversation history) to OpenAI or Anthropic
4. 4LLM → Decision: The model generates not just a response, but also action instructions in JSON format
5. 5Response → Speech: Azure TTS converts the response to speech
6. 6Action → Misty: Movement commands are sent to Misty via the REST API
7. 7Emotion → Display: Misty shows matching facial expressions on its LED display

The Biggest Challenges

1. Speech Recognition in Noisy Environments

Misty's microphones are good, but at trade shows and events it's loud. My solution:

• Noise Gate: Only audio above a certain threshold is processed
• Wake Word Detection: Misty only reacts to direct addressing
• Chunk-based Processing: Audio is processed in segments, not as an endless stream
• Error Tolerance: When transcription is uncertain, Misty asks again

2. Latency – the Enemy of Natural Conversation

The path from microphone → Azure STT → OpenAI → Azure TTS → speaker has latency. My optimizations:

• Streaming Responses: GPT-5 / Claude Opus responds in a stream, TTS starts immediately with the first words
• Caching: Frequent greetings and standard responses are cached locally
• Predictive Loading: While Misty speaks, the next possible response is already being prepared
• In-between Animations: Misty shows "thinking animations" so the wait time feels natural

3. Computer Vision: Recognizing and Remembering Faces

Misty should not only see visitors but recognize them again. For this I use Azure Face API:

• Face Detection: Detection of faces in the camera image
• Face Identification: Comparison with stored faces
• Emotion Detection: Recognition of emotions (smiling, surprised, etc.)
• Personalization: When Misty recognizes someone, he greets them personally: "Hello Marcel, good to see you again!"

4. Autonomous Action – the Hardest Part

The hardest part wasn't speech recognition or vision – it was autonomous decision-making. I wanted Misty to not just react to commands, but to decide on its own what to do next.

My solution: A state machine concept, combined with GPT-5 function calling and Claude Opus:

Misty's States:

• Idle: Misty looks around, reacts to movement
• Greeting: Someone approaches, Misty greets
• Conversation: Active conversation with a person
• Guiding: Misty guides someone to a location
• Presenting: Misty gives a presentation
• Exploring: Misty autonomously explores the environment

GPT-5 decides via function calling which action Misty should perform:

// Example Function Calling Response from GPT-5:
{
  "action": "navigate_to",
  "target": "meeting_room_1",
  "speech": "Follow me to the meeting room!",
  "emotion": "happy",
  "speed": "medium"
}

5. LLM Chaining – Two AI Models Are Better Than One

One of the most exciting techniques in the Misty project is LLM Chaining: the output of one language model is passed directly as input to a second model. Specifically, I use GPT-5 and Claude Opus in a chain:

• GPT-5 generates the initial response or action decision based on context
• Claude Opus validates and refines the response – checking for consistency, safety and tone
• The result is more precise and reliable than with a single model

Why does this matter? A single LLM can hallucinate or produce inconsistent responses. By chaining two different models – one from OpenAI, one from Anthropic – weaknesses cancel each other out. GPT-5 excels at creative generation, Claude Opus at critical review. Together, they deliver results reliable enough for an autonomous robot.

The technical flow:

1. 1Collect context: Sensor data, conversation history, recognized people
2. 2GPT-5 initial response: Generates answer + action instructions (JSON)
3. 3Claude Opus review: Validates the response for safety, appropriateness and consistency
4. 4Final output: The validated response is passed to Misty's speech and movement system

6. Multilingual Support

Misty had to understand and speak German, English, French and Italian – after all, we're in Switzerland. Speech recognition automatically detects the language, and GPT-5 / Claude Opus responds in the same language.

What Misty Can Do Today

After months of development, Misty can:

See

• Recognize faces and personally greet returning visitors
• Identify objects in the environment
• Recognize gestures (waving, pointing)
• Assess the emotions of conversation partners

Hear

• Understand natural speech in 4 languages
• Recognize sounds (clapping, knocking)
• Distinguish voices (who is speaking?)
• Listen in noisy environments

Speak

• Give natural, context-aware responses
• Tell jokes and respond to humor
• Share information about CNEXT and our services
• Make small talk – yes, Misty can talk about the weather

Act Autonomously

• Independently decide whether to greet someone
• Navigate visitors to meeting rooms
• Switch between conversation partners
• Get "bored" and independently explore the environment when no one is around
• Adjust its emotions contextually

What I Learned as a Student

Embodied AI Is More Than Code

The biggest insight: Making a robot intelligent is 30% programming and 70% designing interaction patterns. How long should Misty wait before approaching someone? How does it react when someone walks away mid-conversation? What does it do when it doesn't understand a question?

These aren't technical questions – they're UX questions for the physical world.

The Importance of Error Tolerance

In the software world, you show an error message. When a robot has an error, it just stands there staring at you. Or worse: it says something nonsensical. I learned that graceful degradation is even more important for robots than for websites.

AI Democratizes Robotics

Before OpenAI and Anthropic, it would have taken me years to teach Misty natural conversation. With GPT-5, Claude Opus and Azure AI Services, I was able, as a student – without a robotics degree – to build a robot that delights people. That's the real revolution.

Misty at CNEXT: From Student Project to Showpiece

What started as a student project is now a central part of CNEXT's Embodied AI offering. Misty is:

• Trade show highlight: At events, Misty attracts all eyes
• Customer delighter: Over 10,000 interactions – and people love him
• Technology demonstrator: He shows what's possible with Azure AI and OpenAI
• Conversation starter: Nothing breaks the ice better than a friendly robot

Learn more about our Embodied AI offering on our Embodied AI page. And find the complete Misty project in our project overview.

Conclusion: The Future Is Embodied

We are at the beginning of a revolution. AI agents won't be trapped in screens forever – they will come into the physical world. As robots, as autonomous systems, as companions in everyday life.

And I got to take the first step as a student. A small robot with big eyes became an autonomous AI system that sees, hears, speaks and acts. A student project became a product that delights people.

That is Embodied AI. And at CNEXT, we're just getting started.

Questions about Misty or Embodied AI? Write me: luis.castillo@cnext.ch