DELPHAI: We Built an AI Council That Checks the Math, Argues with Itself, and Knows How to Say No

Written by Matt Graves and Jeff Lynch Introducing DELPHAI: The Award-Winning Reasoning Agent — https://github.com/jlynch160/delphai

“Yes, my team will be ready by the audit date,” is a phrase we often hear from engineering leaders when contracts demand five SC-200 analysts by the end of a quarter, or when compliance frameworks request proof that security training has been completed. However, these assurances tend to rely more on gut feelings rather than solid evidence.

While an impressive-looking 78% average might seem great, it can hide the fact that one analyst may only be performing at 40% in a crucial area that could lead to issues. A plan can appear feasible on paper, but often it means sacrificing evenings and weekends, impacting team morale. There are warning signs, but many tools aim to back up a plan rather than scrutinise it.

This issue is what MajorKey aimed to tackle. When you ask a typical AI assistant, “Can we achieve this?”, it often provides a cheerful roadmap. Yet very few systems assess the actual chances of success, and even fewer will honestly tell a director that the answer is no or reject a plan that pushes team members too far.

For the Microsoft Agents League (Battle #2: Reasoning Agents with Microsoft Foundry), we created DELPHAI — a team of eleven AI agents that debate evidence and calculate feasible outcomes instead of relying solely on intuition. This council provides one of three easy-to-understand verdicts: GO, NEGOTIATE, or NO-GO. Named after the Oracle of Delphi, DELPHAI is designed to answer a key question before an organisation commits: Can we fulfil this promise?

“The model explains the reasoning; it doesn’t generate the figures.” This guiding principle is at the core of every agent in the council.

Single-agent optimism is a limitation. A solo AI assistant lacks anyone to challenge its views, leading it to accept the first feasible plan and refine it. In human decision-making, we counter this by incorporating red teams, sceptics, and stakeholders who can voice concerns. Agentic AI replicates this dynamic effectively at scale.

Vital calculations should fuel the debate. For instance, calculating the probability that at least 5 out of 8 individuals will pass an exam by a specific date is straightforward. Yet, these numbers often don’t reach the meeting where decisions are made, primarily due to the lack of expertise, tools, or time needed for calculation. Equipping an agent to compute probabilities during discussions ensures that conversations are based on evidence rather than assumptions.

AI is increasingly shaping decisions about individuals. Study plans, workload expectations, and readiness evaluations directly impact real people, making safeguards absolutely essential. Responsible AI agents should have the ability to veto a plan or decline an unethical order.

DELPHAI’s readiness console: enter a question (like certification, headcount, or deadline), and various IQ models come online to convene the council.

A manager might ask, “Can we certify 5 out of 8 engineers for SC-200 in 5 weeks?” Eleven advisors analyse the computed evidence in real-time, returning one of three verdicts:

• GO: The calculated forecast meets expectations, and the plan is reasonable. Greenlight the date.
• NEGOTIATE (labelled REVISE in the console): It could work, but only under specific conditions, like extending the timeline or training fewer engineers. Each option comes with a revised probability.
• NO-GO: The honest answer is no, supported by evidence explaining why and what changes would be necessary.

The advisors are designed to engage in healthy disagreement.

Ben, the optimistic planner, approaches each task with a positive perspective. Vera, the sceptic, challenges his assumptions and has veto authority. Maya, the wellbeing advocate, can also decline plans that place excessive demands on team members. Nadia evaluates readiness across different exam domains, while Theo and Omar verify and fact-check claims. Lastly, Dana manages the discussion and reconciles it into a clear verdict along with a path forward.

The outcome yields a decision and a roadmap. Each advisor receives a well-structured weekly study plan, along with calendar-ready .ics blocks and a draft briefing for managers. Individual contributors can also benefit from learner mode, which connects them with Kai, an AI tutor powered by the same decision-making engine.

• 11 reasoning agents
• 3 execution platforms
• 3 IQ classifications
• 8/8 evaluations passed
• 77% of the analyses indicated a NO-GO

An impressive system underpinning this council operates on three execution platforms, connected by a single brain: Foundry Agent Service, where all eleven advisors run as hosted agents debating on a shared platform; the Microsoft Agent Framework, which allows the same council to operate as real agent_framework entities; and a custom streaming orchestrator that supports live demonstrations available through a public link.

Why does this matter? A deterministic reasoning engine acts as the single truth source for all three platforms, meaning you can switch runtimes without altering the verdict logic. The engine makes the decision, and the model narrates it.

DELPHAI is centred around a deterministic reasoning engine. This system assesses each individual’s weekly study capacity based on Work IQ signals, making allowances for real-world factors such as a 22-hour weekly meeting load. It also enforces readiness thresholds per exam domain, ensuring that a good overall average doesn’t mask a vital shortcoming in heavily weighted areas. Finally, it calculates the precise probability of success for the team using a Poisson-binomial distribution, reflecting that each individual has a different likelihood of passing. This results in forecasts based on fact, not guesswork.

FOR TECH-SAVVY READERS

def poisson_binomial_at_least(probs: list[float], k: int) -> float:
    """Exact P(>= k successes) given independent, differing success probabilities."""
    dist = [1.0]  # dist[j] = P(exactly j successes so far)
    for p in probs:
        nxt = [0.0] * (len(dist) + 1)
        for j, pj in enumerate(dist):
            nxt[j] += pj * (1 - p)      # this person misses the date
            nxt[j + 1] += pj * p        # this person certifies in time
        dist = nxt
    return round(sum(dist[k:]), 4)

The engine is fine-tuned against a synthetic dataset that tracks exam outcomes. DELPHAI delves into historical records to determine who passed and who failed, doing so without ever accessing the answer key. This calibration empowers the sceptic and provides a basis for calculating probabilities.

The verdict: Ben estimates a 92% chance of success, Vera a mere 22%, but the council settles on 46%, delivering actionable insights for a positive outcome. Below, you’ll find the IDs of the hosted agents from Foundry Agent Service.

DELPHAI’s purpose is to estimate the feasibility of a project. For example, Ben might predict a 93–96% success rate, assuming no evening commitments and first-attempt passes. However, Vera employs a code-interpreter tool on the Foundry Agent Service runtime and calculates the actual Poisson-binomial based on individual probabilities, arriving at a lower figure of 77%. The optimistic number is quickly crossed out in the DELPHAI interface.

Through DELPHAI, honest and evidence-driven interpretations of decision-making become possible, and the outcomes change significantly. Without DELPHAI, the team likely leans towards GO. With DELPHAI in play, the verdict becomes NEGOTIATE, with real counter-offers — perhaps extending to seven weeks or committing to training four out of eight staff.

A quieter yet crucial moment occurs for our identity and governance clients. Imagine leadership instructing the team to “push their limits” to still meet the hopeful deadline. Maya, the wellbeing advocate, outright declines on the record. As a council member with veto authority, her inputs (like meeting overload and after-hours commitments) are respected and prioritise the team’s needs.

To clarify, team member Priya might seem fine based on the collective average, but she sits at just 58% readiness for Defender for Cloud — below the necessary 75% threshold for an important area. Plus, she has a heavy 22-hour meeting load and a newborn at home. While the averages suggest a GO, the specific evaluations and capacity modelling reveal her actual chances are much slimmer. Rather than a simple no, the council responds with “yes, provided you adjust the timeline or headcount.”

In earlier versions, DELPHAI relied on local keyword searches for advisor grounding. While functional, it felt like a basic solution. We upgraded to Azure AI Search, optimising our knowledge base for easy access through hybrid search, merging BM25 keyword matching with vector search and reciprocal-rank fusion. This semantic layer highlights relevant information that standard keyword searches might miss, ensuring that every advisor’s statement is backed by a clickable reference. Furthermore, a live connection to the Microsoft Learn MCP provides official exam guidance directly from Microsoft.

The same eleven advisors also function as persistent hosted agents on Foundry Agent Service, each boasting its own identity, and on the Microsoft Agent Framework, showcasing the design’s adaptability across different platforms. The debate transpires on a shared thread on the Agent Service, allowing the advisors to interact while documenting all exchanges, including the sceptic’s code-interpreter requests, within the Foundry portal. The clinching moment occurred when Maya rejected an overwork directive on a controlled runtime; this made DELPHAI shift from feeling like merely a demo to a powerful tool.

• Live services tend to fail at inopportune moments. Each grounding path checks its configuration, reverting to a local database when necessary. This local version maintains the same reasoning; only the delivery differs. There’s no demo-mode switch that we can overlook!
• Disparities between two versions emerged. The online dashboard should reflect the Python engine in JavaScript, but inconsistencies arose. We resolved this with checks plus eight automated evaluations that run against both systems. Every build has passed with an 8 out of 8 score.
• Microsoft 365 Agents SDK has pitfalls with API versions. The Agent Framework interacts with the newer Responses API, necessitating a preview version; the standard one returned an unhelpful 400 error. Just one line cost us half a day.
• Windows consoles still present challenges. The az acr build faced a cp1252 encoding issue mid-process, despite the build succeeding server-side. The solution was to enforce UTF-8 encoding before queuing, then check for the finishing tag.
• Drama should not overshadow professionalism. An AI refusing an order may seem overly theatrical; early UI designs leaned too much into this. We opted for a more dignified refusal that presented reasons rather than resembling a sci-fi rebellion.

Although DELPHAI is a hackathon creation and a teaching tool, not yet a commercial product, we have noteworthy insights:

• The tool’s input significantly alters the outcome. Removing the sceptic’s code interpreter results in the council mistakenly accepting a 93% that should realistically be 77%, turning an undeserved GO into a warranted NEGOTIATE. As demonstrated, incorrect assessments are impossible without computational evidence.
• All numbers are traceable. Success probabilities, capacity deductions, and team forecasts arise from the deterministic engine and are supported by source data. The evaluation system replicates the synthetic dataset’s labelled outcomes without ever having seen them: 8 out of 8, every build.
• Three real Microsoft runtimes, with evidence. Hosted agent IDs, the Agent Framework transcripts, and the live analysis showcase debate appear in repository files, and all Agent Service interactions are trackable step-by-step within the Foundry portal.
• Live and offline-ready. The complete streaming council functions on Azure Container Apps at a public URL, with its grounding layer gracefully declining when external services fail, eradicating reliance on conference Wi-Fi or direct connections.

1. Designed disagreement is essential. One prompt issued to all agents means a “council” is merely a single model conversing with itself in various forms.
2. The sceptic needs a calculator, not merely a personality. A doubting agent that only says “I’m not convinced” clutters the conversation. An agent that performs calculations and cites probabilities adds invaluable authority.
3. Refusal is a structural necessity. Maya’s power to decline exists because the architecture mandates that the conductor acknowledges such vetoes. A guardrail that can be overruled is merely a suggestion.
4. Core calculations guided by the narrative model. While the engine computes outcomes, the model explains them, and the boundary between the two is where trust resides. This approach will be our standard for any AI-integrated solution impacting real decisions.

As we advance, our focus will be on enhancing DELPHAI’s connectivity, governance, and broader applications.

• Integrating with Microsoft Graph would allow Work IQ’s adjustments based on actual meeting loads to generate personalised signals.
• Linking Microsoft Learn MCP directly to Agent Service would provide official documentation just one tool call away from each advisor.
• Shifting Fabric IQ’s roles, certifications, skills, and domain relationships into Microsoft Fabric’s semantic layer would produce a governed, searchable knowledge repository.
• Extending the council to cover project delivery, audit readiness, migration timelines, and other commitments where the real issue is whether the promise can be upheld.

Certification readiness often resembles a forecasting issue masked as a motivation challenge. DELPHAI operates with a deterministic engine, equipped with agents that question one another, a sceptic validated by numerical backing, a structured refusal capability, and reasoning that comes with citation. This is the dialogue we engage in with clients every week. The real question has now shifted from whether AI should be part of decisions to whether it can substantiate its contributions, question assumptions, and know when to say no.

DELPHAI began as a hackathon project. The refusal capability is why we kept developing it.

Ask it if your team can be prepared in five weeks, and experience the moment an AI provides you with a truthful answer.

• Try the live council: the DELPHAI demo on Azure Container Apps. Suggested process: Build my path, Whole team, SC-200, 5 of 8, 5 weeks, Convene. No login required.
• Watch the demo: the walkthrough on YouTube
• Access the code: github.com/jlynch160/delphai, which includes the evaluation harness, Foundry IQ indexer, and proof transcripts for all three agent runtimes
• Run it in 60 seconds: clone the repository, run pip install -r requirements.txt, and execute python server.py. Fully offline from the start; Foundry, Azure AI Search, and the Learn MCP activate as you input the keys.
• Customise it: the teams, certifications, and knowledge bases are merely JSON and markdown. Replace them with your own roles and readiness thresholds, allowing the council to deliberate on your deadlines.

Note: DELPHAI was created by MajorKey teammates for the Microsoft Agents League, Battle #2 (Reasoning Agents with Microsoft Foundry). This tool uses synthetic data only, ensuring no real personal information is included in the app or this post. The views expressed belong to the authors.