Adding a Fallback Model to Hermes with Microsoft Foundry

The plan was straightforward. Keep the Bedrock configuration as is and connect Microsoft Foundry as a backup. This way, Hermes will always have an alternative to rely on if the main provider isn’t responsive. Here are a few reasons why I specifically chose Foundry:

• Self-Sufficient Redundancy: If Bedrock is experiencing throttling or is out of its quota, I want Hermes to automatically switch to the backup without me having to intervene.
• Existing Catalogue: I already pay for Foundry, which provides the latest GPT models, along with open-weight and partner models, all in one place. This allows me to choose the best model for each task, rather than being stuck with what one provider has to offer.
• Built-In Enterprise Controls: Features like region pinning, private networking, content filtering, and per-deployment quotas are all in one portal, simplifying security reviews.
• Understanding the System Early: It’s better to understand how Hermes interacts with a provider chain on a calm day rather than amidst a live issue.

In essence, if you’re trying to decide whether to continue reading, here’s the gist: It will take you around thirty minutes if everything goes smoothly. The payment is only for the tokens you use; if the backup isn’t engaged, there’s no charge at all. The outcome? An assistant that continues to function even if your primary provider fails.

Before diving in, you need three essentials: a machine with Hermes installed and configured to use a primary provider, an Azure subscription enabling access to Microsoft Foundry in a deployable region and sufficient quota in that area for deployment.

A significant advantage here is that Foundry deployments are billed per token on a standard pay-as-you-go basis. Therefore, a backup that’s not used doesn’t cost anything beyond the requests it actually handles, making it virtually free until you genuinely need it.

Figure 1: The position of the fallback. Each request routes through the Hermes gateway to the primary provider; it only continues to Microsoft Foundry if that provider is unavailable.

The first phase of this process takes place entirely within the Microsoft Foundry portal and doesn’t involve Hermes yet. Essentially, you need to ensure that your Azure subscription can support a model, and that you have an endpoint and key for Hermes to use later.

Figure 2: Completion of the setup in five steps. The first two occur in the Microsoft Foundry portal (orange), while the remainder take place on the Hermes machine (blue).

To begin, head over to Microsoft Foundry > Build > Models > Deploy > Deploy a base model. If you already possess a fine-tuned model, you can deploy that instead, as it also works seamlessly with Hermes. Make sure to check the region displayed at the top of the portal before confirming, as both model availability and deployment quota can vary by region. Then, go ahead and deploy your chosen model:

For this example, I deployed gpt-5.6-sol, which will be the fallback model for Hermes.

This decision was made intentionally; my primary model on Bedrock is a general-purpose chat model, and a fallback is only effective if the responses it generates feel like a continuation of the same conversation. The gpt-5.6-sol model closely mimics that behaviour, was available in my desired region, and the allotted quota sufficiently supports a day’s normal use. If the fallback surprises you on its first use, it may not be the suitable backup.

Once the deployment is complete, carefully note down two pieces of information: the target endpoint URI and the API key. Save both somewhere secure as you’ll need them for Hermes in the next step. If your organisation rotates keys regularly, choose a key with the longest lifespan allowed, since a backup that relies on an expiring credential will fail when that key becomes inactive.

Figure 3: All the information Hermes will require and its locations before you exit the portal.

With the Foundry aspect sorted, moving forward involves the Hermes CLI entirely. Here’s a vital point: this command pertains to the fallback, not the primary provider, so your existing Bedrock settings will remain intact. None of this can disrupt what is already functioning, making it a safe option to execute on a live setup.

Run the fallback command in Hermes:

When Hermes prompts for the provider to add, select Azure Foundry. Note that in the picker, it still has its old name; it refers to the service known now as Microsoft Foundry.

Paste in the target endpoint URI from your deployment, and authenticate using the API key you copied earlier. Hermes also offers Microsoft Entra ID here, which is preferable if your organisation aims to avoid storing a static key on the machine. If authentication fails at this step, double-check the endpoint before questioning the key; my experience has shown that it’s more common for the endpoint to be incorrect, often due to the deployment name at the URI’s end not matching the one you created.

Once you’re authenticated, Hermes will display the deployments available from your Foundry resource and ask you which ones you wish to use. You can select multiple models, but the ordering is critical: Hermes processes from top to bottom in the list whenever a provider is unavailable. Therefore, treat this selection as a priority order, not just a list.

Figure 4: Real-world interactions of the provider chain. The fallback only earns its keep in the middle two scenarios, highlighting how easy it is to overlook its presence.

At this point, Foundry remains a backup option. I wanted to switch things up and have Foundry as the primary provider while Bedrock takes the fallback position. This adjustment is partly because I prefer keeping everyday traffic within my Azure subscription, and also, I wanted confirmation that the chain functions in both directions.

Figure 5: The arrangement switch in action. No additional features are added or removed; the two providers simply exchange positions.

Unlike some systems, Hermes doesn’t have a specific “promote” command, so this transformation takes a few simple steps rather than a single command:

1. Designate the fallback provider/model as the new primary: hermes model
2. Remove the now duplicated model from the fallback chain: hermes fallback remove
3. Optionally add the former primary model back into the fallback: hermes fallback add
4. Restart the messaging gateway: hermes gateway restart
5. Confirm the changes: hermes status / hermes fallback list

That sequence works fine, and it’s beneficial to understand what’s happening in the background. Yet, since I already had a functioning provider configured, I’d prefer a more seamless way. Fortunately, the tool allows you to converse with the configuration, rather than merely typing commands.

Prompt: “Please set the model I configured on Microsoft Foundry as the primary, and make the Bedrock model the fallback!”

Hermes automatically adjusts the provider chain and confirms the swap, making it much less prone to errors than performing those five commands manually.

Configuration that hasn’t been tested is still just an assumption, so the next step is to verify that Hermes is actually communicating with Foundry.

• Type /model while running Hermes to access the model selector.
  
• You’ll first need to select a provider. Choose the Microsoft Foundry option, and then select the specific deployment underneath it.

• The active model should switch immediately. Send a simple “Hello” message to ensure that the deployment responds, as there is a distinction between appearing in a dropdown and actually processing a request.

A second check is worth doing: use hermes status to verify which provider is currently live, and then use hermes fallback list to ensure that the order of the chain is as you expected. The model selector tells you what you’ve selected; those two commands reveal what Hermes will execute at three in the morning.

Every tutorial has that one part the author conveniently omits. Here’s mine: my initial attempt faced an issue not with the model, but with capacity. I made the mistake of deploying into the closest region out of habit, only to be declined since there was no capacity left at the tier I requested. Although the model was clearly listed in the catalogue, being listed does not guarantee deployability in your region, on your subscription, or under your quota—all three aspects are separate concerns.

Deploying in a different region fixed the issue within a few minutes, but it also resulted in a new endpoint URI, meaning the value pasted into Hermes was now outdated. Running hermes fallback add against the new endpoint resolved it.

The takeaway here is simple but crucial: always check for quota and regional capacity based on your specific subscription before laying out a plan, walkthrough, demo, or production fallback surrounding one deployment. In the Foundry portal, navigating to Management then Quota shows precisely what you’ve been allocated per region and model family, which is the only list that matters.

Figure 6: The four possible issues that arose, or nearly did, and the solutions for each.

On a positive note, thanks to the existence of the fallback chain, when a deployment refused to activate, it didn’t cause total disruption to the assistant. This scenario is the exact situation we aimed to address, and it occurred on day one without necessitating any simulations.

Here’s a summary of everything detailed in this walkthrough:

hermes fallback add: attach a provider to the fallback chain
hermes fallback remove: drop a provider from the chain
hermes fallback list: display the chain in priority order
hermes model: establish the primary model
hermes gateway restart: restart the messaging gateway after any adjustments
hermes status: verify which provider is currently active
/model: switch models during an active session

Integrating Microsoft Foundry as a fallback behind my established Bedrock configuration took about an afternoon, during which most of the time was spent rectifying a regional capacity limit I should have checked beforehand. The actual setup was minimal: deploy a model, copy the endpoint and API key, execute hermes fallback add, authenticate, and choose your deployments. Now, Hermes has the added advantage of not being reliant on just one provider’s uptime.

Three main takeaways from this process should be remembered:

• Verify Quota, Not Only the Catalogue: The Foundry catalogue shows what Microsoft provides, but it doesn’t guarantee availability based on your subscription and regional limitations. Always verify this before proceeding with any designs.
• Deliberately Order Your Fallback Chain: Hermes processes from top to bottom, so the sequence established during setup defines the failover policy you’ll live with. Prioritise models you truly trust at the top of the list.
• Treat the Endpoint as Part of Your Credentials: Redeploying in a new region alters the endpoint URI, and directing a fallback at an outdated endpoint won’t function. Make sure to redo the setup whenever the deployment changes.

Shifting from Bedrock as the primary provider to Foundry has also proven that the chain operates in both directions, which is the crucial point. Provider redundancy is only beneficial if you’ve seen it in action. Next on my agenda is purposefully breaking the primary provider to verify that failover occurs automatically, without needing to input any commands.

If you experiment with this configuration in a different model, region, or provider, I would genuinely appreciate hearing about your experience, especially if your quota management went smoother than mine. Feel free to share in the comments.