Agentic Mentor: A Specification-Driven, Multi-Agent Learning Tool

Instead of just trusting the model to gauge a student’s understanding, we’ve developed a unique pipeline that requires proof of progress: a student must show they grasped the material before moving onto the next step. Additionally, we avoided costly commercial APIs for this teaching approach by creating a system that operates on locally hosted or free-tier models, ensuring accessibility for both students and educational institutions. As part of our collaboration with Microsoft, we’ve made the entire public repository for Agentic Mentor available. With the rise of token-based services in software development, we believe this initiative could significantly influence programs like GitHub Education, where affordable tools for students are absolutely vital.

In this article, we’ll explore the motivations behind this project, the inner workings of Agentic Mentor, and our insights gained from testing it with actual university coursework.

Our project unfolded over three months. Initially, we spent the first weeks conducting background research and gathering requirements. We explored existing literature on AI-assisted learning, specification-driven development, and the nuances of prompts. Additionally, we consulted with our client supervisor from Microsoft, Lee Stott, and our academic supervisors to identify essential functional and non-functional requirements, breaking down the project into manageable components. This exercise clarified what we could realistically achieve within our timeline and highlighted key features that were essential versus those that were merely desirable.

We coordinated the implementation via GitHub, holding daily team meetings to synchronise our parallel tasks, alongside weekly meetings with our supervisors and client to report our progress and ensure we were on the right track. This frequent interaction was critical as the tool’s design was still evolving. It allowed us to address incorrect assumptions quickly—within days rather than weeks. A pivotal decision made from these discussions was shifting Agentic Mentor’s role from an assessment tool to a learning aid. Since the AI cannot accurately read academic briefs as a human would, relying on it for grading could lead to misunderstandings affecting students’ scores. Thus, we redirected our design towards a more supportive learning tool.

In the final month, we focused on completing both user interfaces and evaluating the system. We began with checks against SpecBench for correctness and later tested it using a real master’s coursework piece. Two challenges emerged: our GPU infrastructure went down, forcing us to switch to free-tier cloud models and manage runs around quota limits. Additionally, one of the questions generated by the system was subtly incorrect, an issue we accepted without question due to automation bias—something we had been mindful about throughout the project.

Agentic Mentor is designed as a multi-agent pipeline driven by specifications, featuring four distinct agents: Research, Ingestion, Mentoring, and Viva. Each agent was developed using the Microsoft Agent Framework. This design choice meant that we could equip every phase with tailored tools for their specific tasks instead of relying on a single general-purpose agent. The agents operate sequentially, with each writing its output to disk and passing on file paths to the next stage in the process. This structure allows sessions to be paused and resumed easily. Below, you can see an overview of Agentic Mentor’s architecture.

**Research Agent:** This stage provides the pipeline with relevant context. It leverages GitHub and arXiv MCP servers to collect applicable literature and existing projects. The MCP interface standardises access to sources, allowing additional servers to be linked without overhauling the agent.

**Ingestion Agent:** This component, based on GitHub’s SpecKit, is responsible for transforming the assignment brief and research context into structured specification files. Instead of allowing the model to tackle ambiguities on its own, SpecKit prompts the student directly for clarifications. This keeps students engaged in making design choices rather than letting the model decide for them.

**Mentoring Agent:** In this phase, the student collaborates with the model to build the project step by step, with the agent both coding and explaining its thought process. Progress is capped; students must complete all tasks and pass a short multiple-choice checkpoint demonstrating understanding before progressing further.

**Viva Agent:** Once the implementation wraps up, this agent conducts an interview with the student. It checks their answers against key points established beforehand, providing feedback after each response and saving a full transcript for later use.

**How Microsoft’s tools influenced the project:** The Microsoft Agent Framework was crucial for making our multi-agent design feasible, enabling the construction of four distinct agents that could carry their own tools and remain in communication via a shared layer. Coupled with GitHub SpecKit’s structured methodology for specification-driven development, these resources provided a solid technical foundation that would have been much harder to build from scratch. They were essential for executing the pipeline’s phase-gated, specification-first design successfully rather than leaving it merely as an idea.

This demo video showcases an end-to-end demonstration of Agentic Mentor’s VS Code extension.

https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkKau6IxVq9I%3Ffeature%3Doembed&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkKau6IxVq9I&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkKau6IxVq9I%2Fhqdefault.jpg&type=text%2Fhtml&schema=youtube" allowfullscreen="" style="max-width: 100%

We assessed Agentic Mentor’s performance in two phases: first, a benchmark for basic correctness, followed by a test against a real master’s-level coursework that aligned closely with the intended use of the system.

Using SpecBench, we measured Agentic Mentor’s correctness in coding. SpecBench is a benchmark comprising 30 systems-level programming tasks with established test suites. We executed six of these tasks using a locally hosted Qwen3.6-27B model.

Our findings revealed that tasks with shared structural logic performed well, such as the json_parser achieving a 97.7% pass rate. However, more intricate tasks fell short. For instance, in the crypto_primitives task, hallucinations prevented a viable solution from being produced. This result highlights the limitations of a smaller local model like Qwen3.6-27B. Nevertheless, the success of some simpler tasks indicates that smaller local models do hold potential.

In the second evaluation phase, we examined the system using a real academic assignment focused on Test-Driven Development, which our team had already completed prior to using Agentic Mentor. To mitigate hallucinations and enhance reasoning, we employed Claude Sonnet 5. We combined our assessment as students with insights from an interview with Jens Krinke, the module leader overseeing the coursework.

From our perspective, Agentic Mentor made approaching a significant and loosely specified project far simpler. The checkpoints at each stage demanded active recall, which greatly improved our comprehension of the assignment. It outperformed our original attempt by constructing a synthetic repository with known test-production pairs as clear acceptance criteria, linking test and production files based on their co-occurrence in commit history rather than on naming conventions, and ensuring unit test coverage within a properly structured project.

However, the evaluation of its comprehension was less favourable. Jens rated its understanding on par with the average student, but not better. The system treated commit history as a means to confirm the presence of TDD, while it could merely indicate its absence, and it overlooked an implicit cue related to commit size.

Building Agentic Mentor provided our team with invaluable insights into coding agents, local models, and the practical challenges of applying AI in educational settings. These experiences not only shaped the tool itself but also influenced our understanding of deploying AI in learning environments.

**Understanding the tool’s role:** A standout takeaway was recognising that Agentic Mentor functions best with well-defined, undergraduate-level assignments rather than open-ended research tasks. Our evaluation of SpecBench’s json_parser, which achieved a 97.7% success rate, clearly demonstrated its effectiveness with concrete problems. Conversely, our Test-Driven Development case study revealed that the tool struggled when faced with ambiguous or partially hidden objectives. Regardless, our experience using the tool genuinely enhanced our learning outcomes and even uncovered implementation ideas we hadn’t previously considered, reinforcing our confidence in the potential of agentic learning tools when applied appropriately.

**Navigating the local model trade-offs:** Through our extensive testing of locally hosted models, we learned that there’s a trade-off between accessibility and quality. Local models excelled in simpler tasks, but as complexity increased, issues with correctness and hallucinations became more frequent. This highlighted critical engineering trade-offs: while a completely local, no-cost implementation is theoretically possible, achieving a truly effective model currently relies on access to larger cloud-hosted alternatives.

**Reevaluating the tool’s purpose:** Perhaps our greatest shift in perspective came from rethinking its role. Initially, we imagined Agentic Mentor as an assessment tool capable of measuring students’ understanding for formal grading. However, implementation clarified that an AI system not authored by the assessment brief itself cannot accurately interpret an educator’s intentions. This limitation of natural language interpretation is well documented in the literature we reviewed. Recognising this early enough to change direction was an insightful exercise in engineering judgement, guiding us towards repositioning Agentic Mentor as a learning tool focused on engaging students with agentic AI instead of a grading tool tasked with making decisions it cannot reliably execute.

In summary, our journey in building and assessing Agentic Mentor uncovered three key lessons for educators aiming to integrate AI into their teaching environments:

Our academic evaluation found that AI agents occasionally held technical misconceptions, heightening the risk of automation bias, a significant concern in AI-assisted learning. When an AI responds with absolute certainty, it can easily mislead students. If students accept its outputs without questioning, they may skip over the critical thinking the coursework intends to instill. It’s essential for educational AI to incorporate mechanisms that encourage students to consider the AI’s uncertainties and question its outputs.
The identification of the AI’s technical misunderstandings also underscores the need for caution when using AI for grading or formal assessments. Because the model can misinterpret fundamental concepts in ways similar to a student, it cannot reliably evaluate students’ comprehension. This realization reinforced our decision to shift Agentic Mentor solely towards being a learning aid. AI is immensely useful for fostering student engagement in phased tasks, but it currently lacks the reliability to serve as an independent judge of a student’s true understanding.
As the development of agentic tools transforms software engineering, educators must shift their focus from teaching students to write code manually to teaching them how to engage with coding agents. This includes practices like specification-driven development, which minimises AI errors by clarifying ambiguities in natural language prompts.

The challenges we faced while developing Agentic Mentor highlight two clear paths for further growth, both of which would enhance the system’s reliability as a learning tool:

Currently, Agentic Mentor interprets projects with the same gaps and misconceptions that a typical student might have. Because the system presents its outputs confidently, these gaps can go unchallenged.

We suggest adding a validation agent. This component would interrogate the questions and conclusions generated by the system, introducing a layer of careful consideration and explicitly showcasing assumptions rather than allowing them to be presented as unquestionable facts. The goal would be to minimise automation bias and motivate students to critically engage with the system’s outputs instead of accepting them at face value.

As long as the system’s grasp of an assignment remains limited in the same ways that it might be for a student, it won’t be reliable in pinpointing critical aspects of a task.

Future iterations could incorporate direct input from educators, outlining critical elements and guidance on how to lead students through them. This would necessitate Agentic Mentor evolving from a standalone local tool to one that integrates with educational institutions’ existing platforms. Such integration would provide educators with hands-on control over the system’s actions, boosting its trustworthiness and productivity as a learning resource.

Agentic Mentor was designed to facilitate student learning from their coding assignments rather than merely completing them, embedding the learning process within the system’s structure instead of relying on random coding. We discovered that the model cannot be consistently relied upon for accuracy. Furthermore, utilising smaller local models due to accessibility comes with quality trade-offs. Measuring students’ understanding remains a challenge without extensive user studies. Nonetheless, Agentic Mentor signifies a step toward integrating AI-assisted development in education while prioritising student learning. With appropriate safeguards against automation bias, it can become an accessible tool that aligns the learning experience with the evolving role of AI in software development.

The issue of “one-click” AI code generation in education is an ongoing challenge. However, with Agentic Mentor, our goal is to keep students engaged actively in solving problems rather than taking shortcuts. We encourage you to explore our work and join us in shaping the future of AI in education.

**Run It Your Way:** Execute the orchestrator using the CLI (see the main README), or leverage our custom GUI by running the VS Code Extension (located in the agentic-mentor-extension folder).

Our team behind this project consisted of six members, all Master’s students at UCL, either studying Software Systems Engineering or Artificial Intelligence and Data Engineering.

Mark Connor – Team Leader – Software Engineer

GitHub URL: https://github.com/markjconnor

LinkedIn URL: http://www.linkedin.com/in/mark-connor2003

Alexander Filippov – Software Engineer

GitHub URL: https://github.com/ucabavf

LinkedIn URL: https://www.linkedin.com/in/alexander-f-003a5721b

Weeraya Hew – Software Engineer

GitHub URL: https://github.com/tingwry

LinkedIn URL: https://www.linkedin.com/in/weeraya-hew-924a19261

Tanishka Jaikrishnia – Software Engineer

GitHub URL: https://github.com/tanishkajaikrishnia

LinkedIn URL: https://www.linkedin.com/in/tanishka-jaikrishnia-96b652274/