Back to Blog

    Strategy

    LLM chaining: when a second model helps and how to build it

    Generator, checker, fallback: what we learned about LLM chains on our Misty robot, and how to measure whether the second step pays off.

    February 6, 20265 min read
    Luis Castillo

    Luis Castillo

    Solution Engineer

    luis.castillo@cnext.ch
    CNEXT Microsoft Partner
    In this article

    In LLM chaining, the output of one language model becomes the input of another, for example so that a second model can check an answer, correct it or compare it with a source. That is no proof of correctness: two models can miss the same mistake. Whether the second step is worth it only becomes clear through a measurement that weighs the benefit against latency, cost and complexity.

    What we built on the Misty prototype

    We implemented the pattern on our Misty robot. Misty has to react in real time, and mistakes have physical consequences: a robot that moves in the wrong direction or says something inappropriate is more than a bug.

    An early prototype worked with GPT-5 alone. Three problems showed up: Misty gave out information it could not have known; in noisy surroundings it took background noise for commands; and in long conversations it repeated itself. That is why the chain got a checking step.

    The flow

    1. 1Collect context: Misty's sensors provide recognised faces, spoken words, the current position and the conversation history. These are combined into a structured context.
    2. 2Generate: GPT-5 receives the context and produces a natural-language answer, action instructions in JSON (move, show an emotion, navigate) and an estimate of its confidence.
    3. 3Check: Claude Opus receives this answer together with the original context and checks it against the four points in the table below.
    4. 4Decide: if the checker confirms, the answer is executed. If not, there are three routes: Claude Opus supplies a corrected version, GPT-5 is called again with the feedback, or Misty chooses a safe default action, such as smiling kindly and waiting.
    CheckQuestion for the checker
    SafetyIs the proposed action safe, with no risk of collision and no inappropriate statement?
    ConsistencyDoes the answer fit the conversation so far?
    AppropriatenessDoes the reaction suit the setting, such as a trade fair or an office, a child or an adult?
    CorrectnessAre the facts in the answer right?

    We deliberately chose the generator and checker from two providers, OpenAI and Anthropic. Our reasoning: models from the same provider often share blind spots. That is no guarantee; two different models can still miss the same mistake.

    The final go-ahead does not belong in the model

    Even if the checker assesses safety, a language model should never be the only thing authorising a physical movement. Technical limits, sensors and a safe stop belong in the controller, outside the model. The full pattern therefore has three stages:

    1. 1The generator answers a narrowly defined question using approved sources or, as with Misty, the collected context.
    2. 2The checking step receives the answer, the original question and the sources. It flags unsupported statements or asks for a revision.
    3. 3A deterministic part of the application checks format, permissions and allowed actions. If there is uncertainty, a person is involved or no action is taken.

    Latency, outages and cost

    Latency

    In theory, a chain doubles the response time. We kept it down with three measures:

    • Streaming: GPT-5 streams its answer, and Claude Opus starts checking from the first tokens.
    • Caching: frequent interaction patterns are cached; the chain runs only for new situations.
    • Confidence threshold: if GPT-5 reports high confidence and the request is not critical, Claude Opus is skipped. Self-assessment alone is not enough for this, because language models often give wrong answers with the same confidence as right ones. What matters is that the request is not critical.

    Outages

    • Timeout: each model has a maximum waiting time. If it is exceeded, the fallback applies.
    • Circuit breaker: after several failures, the chain temporarily switches to a single model.
    • Fallback: in the prototype, if the checker failed, GPT-5's answer went through unchecked, with increased logging. For actions with physical consequences that is the wrong way out: if the checker cannot be reached, only the safe default action should remain. The same applies while the circuit breaker has switched to a single model.

    Cost

    Two models cost more than one. Our levers:

    • Not every request needs the full chain: simple greetings go through GPT-5 only.
    • Trim the context: Claude Opus receives only the relevant context, not the whole conversation history.
    • Batch: when several interactions come in quick succession, the checks are batched.

    How to measure the benefit

    Before changing the architecture, put together a test set of typical and difficult cases. Compare a single-model version with the chain on source-backed accuracy, disallowed actions, response time and cost per task. Include outages and contradictory judgements.

    In operation, keep an eye on the checker itself: log how often it changes answers. If it steps in very often, that points to a problem in the generator prompt. If it never steps in, the question is whether you need it for those requests. Generator and checker prompts pursue different goals, so optimise them separately.

    When a chain is worth it and when it is not

    An independent checking step can help with source comparison, structured output or decisions where mistakes are costly. For simple greetings or standard answers, a single call is enough. If you handle many kinds of request, you can build the chain hierarchically: a router decides per task whether a fast model will do or the checking chain is needed. In a parallel variant, two models answer and an aggregator chooses. Medical, legal and financial uses also need expert review and clear accountability; a second model does not replace them.

    Source: NIST AI Risk Management Framework: Measure

    OpenAIClaudeSchweiz
    Teilen:

    AI tools may have supported the creation of this article. CNEXT is responsible for the published content. Product features and prices can change; check the linked primary sources for current details.

    Luis Castillo

    Luis Castillo

    Solution Engineer

    Discuss your AI architecture with CNEXT

    We help you design the generator, checking step and fallback for your use case and measure the benefit.