Misty II provides an HTTP API for movement, display, audio and camera. I connected this interface to Azure Speech, language models from OpenAI and Anthropic and a fixed state machine, so that Misty can listen, answer, greet people and guide them to a place. The biggest hurdles were noise, latency and the autonomous decision logic; the most important lesson concerns interaction design.
How the project started
I'm Luis Castillo, Solution Engineer at CNEXT. Misty was my student project: CNEXT put a Misty II on my desk and said, ‘Make something of it.’
Misty II is a small robot with a display for a face, cameras, microphones and touch sensors, and it can move on its own. Out of the box, it mainly plays pre-programmed animations and responds to simple commands. My goal was a robot that sees, hears, understands, speaks and acts on its own, using language models rather than pre-programmed answers.
The architecture: the brain lives in the cloud
The key design decision: Misty doesn't need a brain on its own hardware. The robot supplies sensor data and carries out commands; cloud services do the understanding and deciding.
| Capability | Service | Task |
|---|---|---|
| Hearing | Azure Speech to Text | Misty's microphones pick up speech, and Azure converts it into text |
| Understanding and deciding | GPT-5 (OpenAI) and Claude Opus (Anthropic) with system prompts | The model decides whether Misty answers, moves or shows an emotion |
| Speaking | OpenAI TTS and Azure Text to Speech | The answer becomes spoken language |
| Seeing | OpenAI Vision and Azure Computer Vision | The camera detects faces, objects and gestures |
| Acting | Misty HTTP API | Misty carries out movements, navigation and facial expressions |
| Chaining | GPT-5, then Claude Opus | The second model reviews the first model's answer |
The model names reflect the state at the time of writing.
The data flow
- 1Hearing: Misty's microphones capture audio.
- 2Audio to text: Azure Speech to Text transcribes it in real time.
- 3Text to the model: The text goes to GPT-5 or Claude Opus together with context: people in view, current position, the conversation so far.
- 4Decision: The model returns an answer plus instructions for action in JSON format.
- 5Speech: Azure Text to Speech turns the answer into speech.
- 6Action: Movement commands go to Misty via the HTTP API.
- 7Emotion: Misty shows a matching facial expression on its display.
Noise: speech recognition at trade fairs and events
Misty's microphones are good, but trade fairs and events are loud. My solution has four parts:
- Noise gate: Only audio above a set threshold is processed.
- Wake word: Misty only reacts when it is addressed directly.
- Processing in segments: Audio is processed in chunks, not as an endless stream.
- Asking back: If the transcription is uncertain, Misty asks again.
Latency: making the wait feel natural
The path from the microphone through speech to text, the language model and text to speech back to the loudspeaker takes time. My optimisations:
- Streaming: The model streams its answer, and speech output starts with the first words.
- Caching: Frequent greetings and standard answers are cached locally.
- Preloading: While Misty is speaking, the next likely answer is already being prepared.
- Thinking animations: Misty shows ‘thinking animations’ so that the wait feels natural.
Autonomy: a state machine with six states
The hardest part was neither speech recognition nor vision, but autonomous decision-making. Misty shouldn't just react to commands; it should decide for itself what to do next. My solution is a state machine combined with function calling:
- Idle: Misty looks around and reacts to movement.
- Greeting: Someone approaches, and Misty greets them.
- Conversation: Misty holds a conversation with a person.
- Guiding: Misty takes someone to a place, such as a meeting room.
- Presenting: Misty gives a presentation.
- Exploring: When nobody is around, Misty explores its surroundings on its own.
The states set the frame; GPT-5 uses function calling to decide which action Misty carries out. A response looks like this, for example:
// Example of a function-calling response from GPT-5
{
"action": "navigate_to",
"target": "meeting_room_1",
"speech": "Follow me to the meeting room!",
"emotion": "happy",
"speed": "medium"
}Two models in a chain
For these decisions, I use LLM chaining: the output of the first model becomes the input of the second.
- 1Collect context: sensor data, conversation history, people in view.
- 2First answer: GPT-5 produces the answer and the action instructions as JSON.
- 3Review: Claude Opus checks the answer for safety, appropriateness, consistency and tone.
- 4Output: The reviewed answer goes to Misty's speech and movement system.
The reason: a single language model can hallucinate or answer inconsistently. A second model from a different provider reviews the answer before Misty speaks or acts on it. This review adds to fixed technical limits but does not replace them (see ‘Limits’ below).
Four languages
Misty understands and speaks German, English, French and Italian; we are in Switzerland, after all. Speech recognition identifies the language automatically, and the model answers in the same language.
What I learned
Interaction design is the bigger job
In my estimate, making a robot intelligent is 30 per cent programming and 70 per cent designing interaction patterns. How long should Misty wait before speaking to someone? How should it react if someone walks away mid-conversation? What does it do when it doesn't understand a question? These aren't technical questions; they're UX questions for the physical world.
Fault tolerance matters more for robots
When software fails, it shows an error message. When a robot fails, it stands there staring at you, or it says something nonsensical. Graceful degradation, meaning controlled behaviour when something goes wrong, is therefore even more important for robots than for websites.
Language models make robotics more accessible
Without the models from OpenAI and Anthropic, it would have taken me years to teach Misty natural conversation. With GPT-5, Claude Opus and Azure AI Services, I was able to do it as a student with no robotics degree.
Limits: safety and data protection
A prototype like Misty needs firm limits before it interacts with people:
- Never execute model output directly. A fixed state machine and a small list of permitted actions are the foundation. A controller validates and constrains every parameter before it executes a command. Movement needs speed and area limits, an emergency stop, obstacle tests and human supervision. A second model reviewing an answer is not a safety certification.
- Control the network. According to the manufacturer's documentation, Misty's HTTP API uses no keys or credentials: any device that can reach its IP address can send commands. The API documentation also says the audiovisual stream is not encrypted. The robot therefore belongs on a controlled network, with no exposed API endpoints and no streaming of conversations to unsecured services.
- Protect face data. Identification with Microsoft Face is a Limited Access feature that requires approval. Under the Swiss Federal Act on Data Protection (FADP), biometric data that uniquely identify a person are sensitive personal data. Recognising visitors therefore requires an assessed purpose, a legal basis, transparent information, consent and technical safeguards. Where processing is likely to result in a high risk, a data protection impact assessment may be required.
- Keep demos data-minimal. For demonstrations, use synthetic content, don't store faces or conversation histories, and give visible notice whenever audio or video is captured. Data flows, providers, retention, deletion and access should be settled before people interact with the robot.
Misty at CNEXT
The student project has become part of our embodied AI offering. At events and trade fairs, Misty shows what Azure AI Services and language models make possible, and it is a good way to start a conversation. You can find out more about the offering on our Embodied AI page, and the Misty project is in our project overview.

