Microsoft's ThinkingBox shows one good run of a business agent is not enough

Microsoft's Copilot Studio team runs AI agents through 507 synthetic business tasks (retail, travel, auto insurance, a neobank and consulting) 20 times each, and checks the database records left behind instead of trusting the agent's claim that it is done. Their earlier results found Claude Opus 5.5 scored 0.66 points higher than Opus 5 on average yet passed exactly the same 241 tasks on all 20 tries, and Kimi-K3 solved 93.89% of tasks at least once but only 68 every time. In this new post the team used the same database checks as a training reward: after 50 training steps, Qwen3.8-27B went from 38 tasks passed on every try to 107, and average success rose from 51.70% to 60.78%. In one hotel task, the base model changed a booking and issued refunds without the required manager approval in 15 of 20 tries; the trained model left the booking and payment untouched in all 20. The training code is open on GitHub under the MIT licence, though the reference run used 24 H100 GPUs.