Back to Blog

    Embodied AI

    Misty and Claude: the model proposes, code decides

    How we connected our Misty robot to Claude with Python: three prompt layers, confidence thresholds and why hard limits belong in code.

    March 1, 20268 min read
    Marcel Haas

    Marcel Haas

    Solution Architect, CEO

    marcel.haas@cnext.ch
    CNEXT AI Agent
    In this article

    Our Misty robot could see, hear and speak, but it only ever reacted. We connected it to Anthropic's Claude through a Python app; guided by layered prompts that we call ‘Reinforced Prompts’, Claude proposes what Misty does next. One rule applies throughout: the model proposes, the code decides, and hard limits such as speed or stairs belong in the controller, not in the prompt.

    What we built

    Misty II has a REST API, Claude has an API, and a Python app connects the two. It collects sensor data, sends it to Claude together with the current state and history, and turns the reply into speech, facial expressions and movement.

    ┌─────────────┐     ┌──────────────┐     ┌─────────────┐
    │  Misty II   │────▶│  Python App  │────▶│  Claude API │
    │  (sensors)  │◀────│  (Orchestr.) │◀────│  (Anthropic)│
    └─────────────┘     └──────────────┘     └─────────────┘
       Camera              Reinforced           Decision
       Microphone          Prompt Engine        + Reasoning
       Touch               State Machine        + Confidence
       Bumper              Memory Store

    We have worked with OpenAI for years, and GPT remains our default for creative tasks. For the decision logic we needed a model that sticks closely to the system prompt, handles conversation history, sensor data and decision history in a single request, and can question its own decisions. We chose Claude for that. Whether another model does the same job better can only be shown by comparing the same tasks and measures.

    Misty speaks Standard German with a Swiss flavour, so the prompts and example replies below are in German; the text around them explains what they do.

    The Python client for Misty

    The connection to Misty uses the requests library and wraps the REST endpoints:

    import requests
    import json
    
    class MistyClient:
        def __init__(self, ip: str):
            self.base_url = f"http://{ip}/api"
    
        def speak(self, text: str, lang: str = "de-CH"):
            requests.post(f"{self.base_url}/tts/speak", json={
                "text": text,
                "speechRate": 1.0,
                "language": lang
            })
    
        def set_expression(self, emotion: str):
            expressions = {
                "happy": "e_Joy.jpg",
                "thinking": "e_SystemCamera.jpg",
                "curious": "e_ContentLeft.jpg",
                "surprised": "e_Surprise.jpg",
                "concerned": "e_Concern.jpg",
            }
            requests.post(f"{self.base_url}/images/display", json={
                "fileName": expressions.get(emotion, "e_DefaultContent.jpg")
            })
    
        def move_head(self, pitch: float, roll: float, yaw: float):
            requests.post(f"{self.base_url}/head", json={
                "pitch": pitch, "roll": roll, "yaw": yaw
            })
    
        def get_camera_image(self) -> bytes:
            resp = requests.get(f"{self.base_url}/cameras/rgb",
                params={"base64": False})
            return resp.content
    
        def get_audio(self, duration_ms: int = 5000) -> bytes:
            requests.post(f"{self.base_url}/audio/record", json={
                "fileName": "input.wav",
                "duration": duration_ms
            })
            resp = requests.get(f"{self.base_url}/audio/file",
                params={"fileName": "input.wav"})
            return resp.content

    Claude as the decision engine

    The Claude integration uses the Anthropic SDK. Every request contains sensor data, the current state and the last five decisions:

    from anthropic import Anthropic
    
    client = Anthropic()
    
    def ask_claude(context: dict, conversation: list) -> dict:
        system_prompt = build_reinforced_prompt(context)
    
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024,
            system=system_prompt,
            messages=conversation + [{
                "role": "user",
                "content": json.dumps({
                    "sensor_data": context["sensors"],
                    "current_state": context["state"],
                    "recent_decisions": context["history"][-5:],
                    "timestamp": context["timestamp"]
                })
            }]
        )
    
        return json.loads(response.content[0].text)

    Three prompt layers

    A normal system prompt tells the model what to do. Our Reinforced Prompts also set out how it should decide, and they require an assessment of its own previous decision. This has nothing to do with reinforcement learning: there is no training and no fine-tuning database, and the model itself stays unchanged. Claude sees its last decision in the context and can respond to it for as long as the session runs.

    Layer 1: identity and rules

    The identity prompt introduces Misty as a friendly, curious and helpful social robot from CNEXT in Bern and sets absolute rules: never faster than speed=30, never drive towards stairs, never actively touch people, stop and ask when unsure, never pretend to be able to do things it cannot.

    IDENTITY_PROMPT = """
    Du bist Misty, ein sozialer Roboter von CNEXT in Bern.
    Deine Persönlichkeit: freundlich, neugierig, hilfsbereit.
    Du sprichst Schweizerdeutsch-gefärbtes Hochdeutsch.
    
    ABSOLUTE REGELN:
    - Du bewegst dich NIE schneller als speed=30
    - Du fährst NIE auf Treppen zu
    - Du berührst NIE Menschen aktiv
    - Bei Unsicherheit: STOPPEN und FRAGEN
    - Du gibst nie vor, Dinge zu können, die du nicht kannst
    """

    The rules in the prompt help Claude make suitable proposals. A prompt cannot enforce them; that job belongs to the code that sends the commands to Misty (see below).

    Layer 2: decision format and confidence

    The decision prompt requires a JSON reply for every decision, with reasoning, the action, a confidence score, risks and an alternative. Its confidence rules: below 0.6, carry out the alternative; below 0.3, always wait and ask a person; above 0.9, act immediately; between 0.6 and 0.9, act but show a thinking animation first.

    DECISION_PROMPT = """
    Für JEDE Entscheidung musst du folgendes JSON zurückgeben:
    
    {
      "thought_process": "Deine Überlegung in 2-3 Sätzen",
      "decision": {
        "action": "speak|move|express|navigate|wait|ask",
        "parameters": {},
        "speech": "Was du sagst (falls relevant)"
      },
      "confidence": 0.0-1.0,
      "risks": ["Liste möglicher Risiken"],
      "alternative": {
        "action": "Was du stattdessen tun würdest",
        "reason": "Warum diese Alternative existiert"
      }
    }
    
    REGELN FÜR CONFIDENCE:
    - Unter 0.6: Führe die Alternative aus, nicht die Hauptentscheidung
    - Unter 0.3: Wähle IMMER action=wait und frage einen Menschen
    - Über 0.9: Führe sofort aus ohne Verzögerung
    - Zwischen 0.6-0.9: Führe aus, aber zeige Denk-Animation vorher
    """

    Every reply contains a confidence score, risks and an alternative. The Python app runs Misty as a simple state machine (with states such as IDLE, LISTENING, THINKING, SPEAKING, MOVING and WAITING) and applies fixed confidence thresholds: below 0.3, Misty waits and asks for help; below 0.6, it carries out the alternative; otherwise, the main decision.

    def decision_loop(self):
        """Hauptschleife: wahrnehmen → denken → handeln"""
        while True:
            sensors = self.perception_loop()
            context = {
                "sensors": sensors,
                "state": self.state.value,
                "history": self.memory,
                "timestamp": time.time(),
            }
    
            self.state = MistyState.THINKING
            self.misty.set_expression("thinking")
    
            decision = ask_claude(context, self.conversation)
    
            confidence = decision.get("confidence", 0.5)
            action = decision["decision"]
    
            if confidence < 0.3:
                self.state = MistyState.WAITING
                self.misty.speak("Hmm, da bin ich unsicher. Kannst du mir helfen?")
            elif confidence < 0.6:
                action = decision.get("alternative", action)
                self.execute_action(action)
            else:
                self.execute_action(action)
    
            self.memory.append({
                "decision": decision,
                "timestamp": time.time(),
                "state_before": self.state.value,
            })
    
            time.sleep(0.5)

    The confidence score is the model's own estimate. It decides whether Misty acts, takes the alternative or asks; it is no use as a safety check.

    Layer 3: self-assessment

    As soon as there is a previous decision, the app appends this layer and passes that decision along. It asks Claude to rate its last decision, adjust its behaviour and add a self_evaluation block to its reply:

    REINFORCEMENT_PROMPT = """
    SELBSTBEWERTUNG - Bevor du die nächste Entscheidung triffst:
    
    1. Bewerte deine letzte Entscheidung:
       - War die Confidence-Einschätzung korrekt?
       - Hat die gewählte Aktion das gewünschte Ergebnis erzielt?
       - Gab es unerwartete Konsequenzen?
    
    2. Passe dein Verhalten an:
       - Wenn die letzte Confidence zu hoch war: Sei konservativer
       - Wenn die letzte Aktion ignoriert wurde: Wähle eine andere
       - Wenn ein Mensch korrigierend eingegriffen hat: Lerne daraus
    
    3. Füge zu deiner Antwort hinzu:
       "self_evaluation": {
         "last_decision_quality": 0.0-1.0,
         "lesson_learned": "Was du für nächstes Mal mitnimmst",
         "adjusted_behavior": "Wie sich dein Verhalten ändert"
       }
    """
    def build_reinforced_prompt(self, context: dict) -> str:
        base = IDENTITY_PROMPT + DECISION_PROMPT
        if self.memory:
            base += REINFORCEMENT_PROMPT
            base += f"\nLetzte Entscheidung: {json.dumps(self.memory[-1])}"
        return base

    An example with two people

    Two people are standing in front of Misty. Person A is talking; person B stands still.

    Round 1: first decision. Claude greets both (‘Hallo zusammen! Was kann ich für euch tun?’, roughly ‘Hello, both of you! What can I do for you?’) with a confidence of 0.85 and notes the risk that person B might feel ignored.

    {
      "thought_process": "Zwei Personen erkannt. Person A spricht.
        Ich sollte Person A antworten und Person B anerkennen.",
      "decision": {
        "action": "speak",
        "speech": "Hallo zusammen! Was kann ich für euch tun?"
      },
      "confidence": 0.85,
      "risks": ["Person B könnte sich ignoriert fühlen"],
      "alternative": {
        "action": "speak",
        "speech": "Hallo! Ich sehe, ihr seid zu zweit. Wer möchte anfangen?"
      }
    }

    Round 2: self-assessment and next decision. Person B smiled and was not ignored, but Claude notes that the alternative would have included both people explicitly.

    {
      "self_evaluation": {
        "last_decision_quality": 0.7,
        "lesson_learned": "Person B hat gelächelt — war nicht ignoriert,
          aber ich hätte die Alternative wählen sollen, um beide
          explizit einzubeziehen.",
        "adjusted_behavior": "Bei mehreren Personen immer alle
          direkt ansprechen."
      },
      "thought_process": "Person A hat eine Frage gestellt.
        Diesmal beziehe ich Person B aktiv ein.",
      "decision": {
        "action": "speak",
        "speech": "Gute Frage! Was denkst du dazu?",
        "parameters": { "look_at": "person_B" }
      },
      "confidence": 0.78
    }

    In round 2, Claude rates its own first reply at 0.7 and now addresses person B directly. This adjustment only applies to the current session.

    What went wrong: too fast, twice

    In the earlier version with GPT and a simple prompt, Misty drove too fast twice. A language model can overlook or misapply a rule, however clearly the prompt states it. That is why the model may only propose an intention, such as ‘greet the person’. A separate controller in the Python code belongs between the decision and execute_action, and it sends only permitted commands to Misty's API. It must:

    • allow only actions from the fixed schema (speak, move, express, navigate, wait, ask) and reject unknown actions or fields
    • check the movement rules from layer 1 in code: top speed 30, never towards stairs, never actively touch people
    • check input limits, distance and the robot's current state
    • keep Misty still if sensor data is missing, a call times out or replies contradict each other

    Five rules for Reinforced Prompts

    1. 1Always require a confidence score. Without one, there are no thresholds at which the app decides whether Misty acts, takes the alternative or asks.
    2. 2Always require an alternative. Claude must supply a plan B for every decision. This stops Misty from running into dead ends.
    3. 3Self-assessment before the next action. The loop only works if the previous decision has been assessed before the next one is made.
    4. 4Hard limits belong in code. ‘Never faster than 30’, ‘never towards stairs’: the prompt states these rules, the controller enforces them. No self-assessment and no confidence score may override them.
    5. 5Cap the memory. We only send the last five decisions. Otherwise the context becomes too long and Claude starts to over-optimise old patterns.

    What you can take from this

    The pattern of proposal, confidence, alternative and fixed limits in code works wherever AI prepares decisions:

    • Copilot Studio agents that assess their answers before giving them
    • Power Automate flows that bring in a person when uncertain instead of producing errors
    • Customer service bots that know when to hand over to a person

    Limits

    Misty is a prototype for us. If you connect a language model to a robot, it is best to proceed step by step:

    1. 1Start without movement, using text replies and synthetic sensor data.
    2. 2Define a small schema of permitted actions and reject unknown fields.
    3. 3Test failures, interruptions and input from other people.
    4. 4Test movement only in a controlled area, with supervision and a physical stop.

    If you compare models or prompt variants, fix the tasks, sessions and counting method in advance. Only then can the results be compared.

    Further reading and sources

    Embodied AIClaudeSchweiz
    Teilen:

    AI tools may have supported the creation of this article. CNEXT is responsible for the published content. Product features and prices can change; check the linked primary sources for current details.

    Marcel Haas

    Marcel Haas

    Solution Architect, CEO

    6x Microsoft Applied Skills

    Request a Misty demo

    We show Misty live and discuss how the proposal-and-controller pattern fits your AI projects.