Our Misty robot could already see, hear, and speak. But it only ever reacted — never decided on its own. We changed that. We connected Misty directly to Claude from Anthropic via Python and taught it to make autonomous decisions using reinforced prompts. Here is exactly how we did it.

Why Claude Instead of GPT?

We have worked with OpenAI for years. For Misty, we deliberately chose Claude — for three reasons:

• Instruction adherence — Claude follows system prompts more precisely. For a robot that acts autonomously, that is not nice-to-have — it is safety-critical.
• Longer context — Claude can process the entire conversation history, sensor data, and decision history in a single request.
• Better self-reflection — Claude can question its own decisions. We use that for the reinforcement loop.

GPT remains our go-to for creative tasks. For autonomous decision logic, Claude is better.

The Architecture: Python as Bridge

Misty has a REST API. Claude has an API. Python connects both. The architecture is simple — and that is exactly why it works:

+---------------+     +----------------+     +---------------+
|  Misty II     |---->|  Python App    |---->|  Claude API   |
|  (Sensors)    |<----|  (Orchestr.)   |<----|  (Anthropic)  |
+---------------+     +----------------+     +---------------+
   Camera              Reinforced           Decision
   Microphone          Prompt Engine        + Reasoning
   Touch               State Machine        + Confidence
   Bumper              Memory Store

The Python Client for Misty

First, the connection to Misty. We use the requests library and wrap the REST endpoints:

import requests
import json

class MistyClient:
    def __init__(self, ip: str):
        self.base_url = f"http://{ip}/api"

    def speak(self, text: str, lang: str = "en-US"):
        requests.post(f"{self.base_url}/tts/speak", json={
            "text": text,
            "speechRate": 1.0,
            "language": lang
        })

    def set_expression(self, emotion: str):
        expressions = {
            "happy": "e_Joy.jpg",
            "thinking": "e_SystemCamera.jpg",
            "curious": "e_ContentLeft.jpg",
            "surprised": "e_Surprise.jpg",
            "concerned": "e_Concern.jpg",
        }
        requests.post(f"{self.base_url}/images/display", json={
            "fileName": expressions.get(emotion, "e_DefaultContent.jpg")
        })

    def get_camera_image(self) -> bytes:
        resp = requests.get(f"{self.base_url}/cameras/rgb",
            params={"base64": False})
        return resp.content

Reinforced Prompts: The Brain Behind the Brain

Here is where it gets interesting. A normal system prompt tells Claude what to do. A reinforced prompt tells Claude how to think — and forces it to evaluate and improve its own decisions.

The concept has three layers:

Layer 1: Base Prompt (Identity & Rules)

IDENTITY_PROMPT = """
You are Misty, a social robot by CNEXT in Bern, Switzerland.
Your personality: friendly, curious, helpful.

ABSOLUTE RULES:
- NEVER move faster than speed=30
- NEVER drive toward stairs
- NEVER actively touch humans
- When uncertain: STOP and ASK
- Never pretend to have capabilities you do not have
"""

Layer 2: Decision Prompt (How to Think)

DECISION_PROMPT = """
For EVERY decision you must return this JSON:

{
  "thought_process": "Your reasoning in 2-3 sentences",
  "decision": {
    "action": "speak|move|express|navigate|wait|ask",
    "parameters": {},
    "speech": "What you say (if relevant)"
  },
  "confidence": 0.0-1.0,
  "risks": ["List of possible risks"],
  "alternative": {
    "action": "What you would do instead",
    "reason": "Why this alternative exists"
  }
}

CONFIDENCE RULES:
- Below 0.6: Execute the alternative, not the main decision
- Below 0.3: ALWAYS choose action=wait and ask a human
- Above 0.9: Execute immediately without delay
- Between 0.6-0.9: Execute, but show thinking animation first
"""

Layer 3: Reinforcement Loop (Learn from Experience)

This is where the real magic happens. After every decision, Claude evaluates its own previous decision:

REINFORCEMENT_PROMPT = """
SELF-EVALUATION - Before making your next decision:

1. Evaluate your last decision:
   - Was the confidence estimate correct?
   - Did the chosen action achieve the desired result?
   - Were there unexpected consequences?

2. Adjust your behavior:
   - If last confidence was too high: Be more conservative
   - If last action was ignored: Choose a different one
   - If a human intervened to correct: Learn from it

3. Add to your response:
   "self_evaluation": {
     "last_decision_quality": 0.0-1.0,
     "lesson_learned": "What you take away for next time",
     "adjusted_behavior": "How your behavior changes"
   }
"""

Results: What Changed

Since switching Misty to Claude + Reinforced Prompts:

Metric	Before (GPT, simple prompt)	After (Claude, Reinforced)
Correct decisions	~68%	~89%
Uncertain situations detected	~40%	~91%
Human corrections needed	~12/hour	~3/hour
Conversation naturalness	OK	Much better
Safety incidents	2 (drove too fast)	0

The biggest win: Misty now recognizes when it is uncertain. Instead of just doing something, it asks. That makes the difference between a cool demo and a robot you can trust.

What You Can Take Away

Reinforced prompts are not just for robots. The concept works anywhere AI needs to make decisions:

• Copilot Studio Agents — Agents that evaluate their own answers before presenting them to users
• Automated Workflows — Power Automate flows that involve a human when uncertain instead of producing errors
• Customer Service Bots — Chatbots that know when to hand off to a human

The future does not belong to AI that knows everything. It belongs to AI that knows what it does not know.

Further Reading

Internal:

• Luis' Misty Article: OpenAI Meets Robotics
• LLM Chaining: Multi-Model Architecture
• CNEXT Embodied AI

External:

• Anthropic Claude Documentation
• Misty Robotics API Docs