Test report
SampleModel comparison on three tasks
- Customer
- Muster AG, Bern (fictional)
- Tasks
- Summarising contracts, questions on internal rules, code review
- Tested
- 14 to 20 October 2026
- Test data
- Anonymised documents and a test repository
- 4models compared
- 3tasks
- 120test cases
- 2models recommended
Summary
We tested two cloud models and two open models on three Muster AG tasks, with 120 test cases from anonymised documents. For contracts and code we recommend a cloud model with governed data processing. For questions on internal rules, an open model on your own hardware answered just as accurately and stays inside your network.
| Cases | Task and result | Recommendation |
|---|---|---|
| 40 | Summarising contracts: cloud model A found every notice period, open model C missed two in appendices. | Cloud model A |
| 50 | Questions on internal rules: all four answered with a source, open model C as accurately as the cloud models. | Open model C, in your network |
| 30 | Code review: cloud models A and B found the planted bugs, the open models about half of them. | Cloud model A or B |
To settle before use
- Legal reviews the data processing agreement for the cloud model.
- The open model needs a server with a GPU; IT has the quote.
Next steps
With your decision we set up the chosen models and rerun the test cases after every model change. The real report names each model and version.

