Paper: How utilizing the Microsoft Copilot harness in Copilot Studio enabled complex task execution

Overview:

The Standard harness of Copilot Studio is designed for seamless conversation. It allows for multiple isolated model calls in a single interaction, provides quick responses, and gives control back to the user as soon as input is needed. However, many business tasks go beyond simple question-and-answer formats. They typically span numerous interactions, involve large datasets, and require coding, file management, and multiple retries throughout the process. To address these complexities, we adopted the Microsoft Copilot harness. This system employs a central-agent loop that operates within a secure environment, offering command line and file tools, as well as the capability to handle substantial data sets on disk for iterative reasoning.

This document compares the two frameworks and highlights the differences. When evaluating the same tasks using the same models, we’ve found that the Microsoft Copilot harness makes twice as many model calls per task. Thanks to effective caching (averaging a 40 percentage point increase), it reduces cost-weighted token consumption by 15 to 21% while improving the evaluation pass rate by 10.8 points on GPT-5.5 Chat and 22.1 points on Opus 4.8. Additionally, the latency per call decreases by 10% to 36%. In real-world applications, agents using the Microsoft Copilot harness are capable of handling longer tasks with more frequent tool calls. They leverage the built-in tools of the container in over half of all interactions and continue operating after a tool call failure 84% to 91% of the time, in stark contrast to the Standard harness, which succeeds in under 7% of cases. In a benchmark for business processes, the Microsoft Copilot harness achieved a score of 65%, while a prominent competitor scored only 36%. Moreover, iterative tool behaviour in production has remarkably improved the success rate of the Excel Online connector from 85.6% to 96.3% with a simple telemetry-led tuning adjustment, requiring no maker input.

Check out our blog here

To access the complete research paper, please click on the PDF attachment below.