Building a Privacy-Aware Personal Agent with Microsoft Foundry Local

The aim of this system was to maintain planning and private workspace access on the device while allowing certain carefully selected subtasks to access cloud support. The cloud would only see a limited version of the evidence, while the original data remained secure on the local device, retaining control over the final output.

I explored three types of tasks:

• Resolving discrepancies in current contracts when signed records, previous agreements, and drafts conflict.
• Creating a briefing by linking a calendar event to relevant emails and documents.
• Verifying archive conditions and documenting notes locally, as permitted by the task.

These tasks require more than just generating credible text. The system must identify the correct information source, maintain citations, adhere to permissions, and sometimes deliver consistent results. Consequently, this project combined an agent architecture with an evaluation framework to verify both the quality of answers and the execution process.

The system is based on a structured workflow that includes coordination, planning, execution, finalisation, and validation. To manage this, I used the Microsoft Agent Framework for orchestration, while Foundry Local managed the language model locally.

The local coordinator determines if a request needs an immediate response, clarification, or access to the workspace. For workspace requests, the planner generates formatted subtasks along with their dependencies. The executor uses local tools to gather evidence and perform permitted actions. Some steps rely on a model, while others, like data retrieval and policy checks, use deterministic functions.

A key choice was using deterministic BM25-style document retrieval. This method divides documents into overlapping sections and ensures that retrieved evidence comes with precise file and line citations. This approach not only makes retrieval reproducible but also facilitates provenance checks, eliminating the need for an embedding model as a variable in this study.

I assessed models for various functions, including planning, structured tool usage, sanitisation, and finalisation, all while considering latency and memory requirements. The frozen experiment utilised Qwen 2.5 7B via Foundry Local and DeepSeek V4 Pro for cloud reasoning, with thinking disabled. A single local model was used for multiple roles, simplifying the operational complexity associated with maintaining several loaded models.

The next challenge was integrating cloud assistance without compromising the workspace.

Figure 1. The privacy workflow ensures the original request, raw tool results, and placeholder mapping are kept on-device. “Untrusted” marks the cloud as outside the trust boundary of this project. This diagram summarises the workflow, showing that execution and optional refinement occur for each subtask.

Before making a cloud request, the system sanitises the instructions and local results of a subtask. Presidio recognisers identify sensitive information, and an optional local model enhances this with context-specific detections. Recognised information is transformed into stable typed placeholders, like and . This joint processing of instructions and context helps maintain consistency in replacements.

The original values mapping stays local. A separate CloudTaskEnvelope only includes a task identifier, task type, redacted instructions, context, and behavioural constraints like preserving citations. It omits raw tool objects and placeholder mapping.

This separation is crucial, as the cloud adapter only receives the envelope and not the entire local execution state.

The router then conducts deterministic privacy checks. If there’s a sanitisation error, critical secret, unavailable cloud service, or an exhausted escalation budget, the subtask remains local. Only after these checks is it determined whether cloud refinement is justified. The system uses a fixed threshold of 0.5 based on a heuristic score derived from local confidence and the characteristics of the task or evidence. This score is not a calibrated probability.

Once a cloud call is made, the system scans the response for issues such as leaks of raw values, fictitious placeholders, and detected personal data. Additional checks ensure that required fields and existing citations are not lost. If the call fails or the refinement is rejected, the system defaults to the original local result. Accepted refinements are combined with the local evidence, and any permitted placeholder restorations occur on the device.

This process is completed before dependent subtasks can proceed, meaning cloud assistance can influence subsequent reasoning while workspace tools stay under local control.

The implementation is carried out in Python, with each key component serving a specific function:

The file tools enforce strict boundaries. They reject paths outside the permitted workspace and only allow writing new notes to a designated output folder, ensuring that existing files remain untouched. Operational traces are stripped of prompts, document content, tool results, and placeholder mappings, extending the design’s data-boundary philosophy into debugging and monitoring.

To evaluate performance, I created simulated workspaces annotated with known facts and sensitive-value markers. This enabled me to verify what crossed into the cloud without exposing any real personal data.

The benchmark included 27 tasks, divided into three families and three levels of complexity, with three instances for each family-level combination. Complexity escalated with additional sources, distractions, reasoning steps, and conflicts. Each task was executed three times under five different conditions, resulting in 405 total executions.

The three main conditions were Local, Hybrid, and Structured-cloud. Local did not utilise any cloud calls. Hybrid selectively escalated certain sanitised subtasks. Structured-cloud followed a comprehensive structured workflow that incorporated cloud models in its cognitive roles and accessed raw synthetic evidence, acting as a reference ceiling within this study.

Additionally, two diagnostic conditions were introduced for context. Cloud-escalated sent every privacy-compliant subtask for cloud refinement. Direct-cloud leveraged an autonomous cloud tool loop instead of the fixed structured workflow.

These represent comparisons of entire systems with varied model placements, contexts, orchestration methods, and privacy controls. The experiment does not isolate model capabilities independently.

The primary evaluator examined facts, conflict resolution, citations, outcomes, and instruction compliance, assigning a quality score from 0 to 100, with a stricter pass rate requiring full marks for each applicable item. The checks for privacy and execution policy were kept separate from the answer quality assessment. Azure AI Evaluation was also employed for a supplementary semantic review of the provided answers.

Quality scores are calculated by averaging repetitions within each task, then averaging across the 27 tasks. The payload is determined as an arithmetic mean. The Structured-cloud condition intentionally transmits raw synthetic evidence, failing to meet the study’s raw-data deployment limitation.

Hybrid surpassed Local by 3.24 points, with a 95% confidence interval for task-bootstrap of 0.84–5.78 points. It successfully narrowed the quality gap with Structured-cloud by 60.72% while transmitting just 18.89% of Structured-cloud’s outbound payload volume. Here, payload volume is measured in bytes sent, not percentage of privacy maintained.

The increase in quality was particularly pronounced with more complex tasks: 1.23 points for Easy, 3.48 for Medium, and 5.02 for Hard. Individual task outcomes varied, with Hybrid outperforming nine tasks, tying in sixteen, and scoring lower in two.

Figure 2. Quality and outbound payload across the five experiment conditions. Filled markers indicate conditions that adhered to the declared raw-data rule; hollow markers highlight references that crossed that threshold. The vertical axis begins at 91.5, and kB signifies 1,000 bytes. The connecting line denotes quality and payload only—not latency or other deployment protocols.

However, this improvement did come at a noticeable cost in terms of latency: Hybrid’s median was 93.33 seconds, compared to 63.10 seconds for Local. The mean cost of cloud inference was around US$0.000232 per task during the experiment, in contrast to US$0.000758 for Structured-cloud. These figures do not account for local hardware and energy expenses and are based on historical data rather than current pricing estimates.

Cloud-escalated achieved a score of 95.44, closely resembling Hybrid, but with a higher data volume sent. Direct-cloud had a score of 92.66. These results suggest that both orchestration and the evidence available to a model should be considered, in addition to cloud access. The negligible difference between Hybrid and Cloud-escalated does not establish a universal advantage for the routing policy.

I obtained non-empty answers from all 405 executions. Local, Hybrid, and Cloud-escalated adhered to the established raw-data rule, with no recorded leakage of raw values or entities in the assessed cloud outputs. However, privacy success did not ensure that every execution passed all checks: Hybrid succeeded in meeting combined deployment requirements in 78 out of 81 executions, indicating that separate tool-policy and security failures did happen.

Azure’s evaluation largely corroborated the quality hierarchy observed in the primary conditions, as it assessed the same answers, affirming agreement rather than independent verification.

This project has reshaped my perspective on evaluating an assistant’s performance. A fluent answer represents only part of the story—I also need to know which sources were used, whether permissions were followed, and if actions were properly executed. Keeping these evaluations distinct allows for clearer interpretation of failures.

I have also learned to make privacy standards openly testable within the code. A designated cloud contract, local state restoration, and deterministic routing checks provided concrete interfaces for testing. However, it’s important to emphasise that achieving a perfect score in synthetic leakage evaluation does not prove resistance to re-identification, cumulative disclosures, or all potential attacks.

Reproducibility became a key focus in this engineering effort. Saved answers, versioned evaluations, corpus hashes, and resetting workspace states enabled repeated analyses without needing to call upon the models again. Examining inconsistencies between deterministic scoring and semantic assessments further underscored the importance of looking into individual records behind overall scores.

This was a solo research project where I developed the implementation, evaluation, and conducted all analyses. Lee Stott from Microsoft and Gerard Gorman from Imperial College London supervised it, and I’m particularly grateful to Lee for his guidance throughout.

AI coding assistants aided in development and reviews, as disclosed in my dissertation. I verified their contributions while retaining responsibility for the design, implementation, and interpretation of the project.

The current evidence is drawn from a synthetic development corpus and encompasses three task families, one device, and a selected local–cloud model combination. There was no study involving real users, and the three repetitions provide limited insight into run-to-run variability.

Looking ahead, I plan to evaluate the system against newly generated, unseen tasks and in closely controlled real-workspace environments. Privacy assessments should also investigate adversarial inputs, re-identification risks, and cumulative disclosures over multiple interactions. Finally, testing across various model pairs and hardware configurations could establish whether calibrated routing and a more efficient local pipeline can sustain quality gains without increasing wait times.

Potential applications include assisting with meeting preparations and document-based tasks in scenarios where raw documents must stay on the device. Demonstrating effectiveness in these contexts will necessitate broader testing and feedback regarding latency, refusals, and restored answers.

This project has demonstrated that selective and sanitised cloud refinement can enhance a local personal agent within a specified benchmark. The significant engineering outcome is a clear boundary that is inspectable, alongside metrics that outline its quality and performance trade-offs.

If you’re venturing into a similar design, start with a compact fictional workspace and a task with a verifiable correct outcome. Establish a local benchmark, scrutinise the exact data proposed for cloud transmission, and gauge the results after introducing optional refinement.

Consider these resources that cover the technologies employed in the project:

Which part of your agent’s workflow truly necessitates cloud reasoning, and what evidence would justify sending it? Please share your use case or any questions in the comments!