Microsoft Foundry Guardrails Getting in Your Way? Here’s How to Dial Them Down

Let me share my recent experience with you. I returned from a trip with a mountain of footage saved on my laptop. Being my usual self (let’s say efficient), I opened OpenCode and requested GPT-6 Astra, which runs on Microsoft Foundry, to create a travel video for me. I wanted something simple: under two minutes, good lighting, smooth transitions, with no embarrassing effects. After a quick quality check, I expected the final product saved neatly in its own folder. Sounds straightforward, right? Well, guess again.

This is the surprise I received:

“Image processing blocked due to content policy violation.” Seriously? It’s just vacation footage! There’s nothing even slightly inappropriate in it. This was clearly a false positive caused by the default guardrails, but the good news is that it’s quite fixable. Let me guide you through the solution.

First, a bit of context: I have two models running on Microsoft Foundry—gpt-6-astra and gpt-6-sol—both using Microsoft’s standard guardrails.

To tackle this, head over to the Microsoft Foundry portal and click on Build. Then, select Guardrails from the left menu. You’ll come across three default guardrails from Microsoft: Microsoft.Default, Microsoft.DefaultV2, and Microsoft.MAIDefault.

The guardrail Microsoft.DefaultV2 is the one that comes with every new deployment, which is why both models are running it. Here’s the catch: you can’t modify or delete these Microsoft-set defaults. So, instead of tweaking them, we’ll create our own guardrail to replace it.

Click on the “Create” button to access the Create guardrail page, where you’ll find all the settings:

Each control consists of three parts: the type of risk, intervention point (whether it checks the input, the output, or both), and the action taken. To ease up those restrictions, switch the action from Block to Annotate. By choosing Annotate, Foundry will still review the content and label it in the API response but will no longer outright reject your request.

Before you start flipping switches, it’s important to understand what each setting does and what you might risk by loosening the controls. Here’s a quick overview:

• Jailbreak: This detects prompts trying to trick the model into disregarding its rules. When set to Block, such prompts will be outright rejected. Changing it to Annotate means they’ll pass with a flag. If you turn it off completely, you’re trusting the model’s training to keep it in line.
• Network (egress rules): This determines which outbound calls a hosted agent can make. However, it only impacts hosted agents, so it won’t affect a plain model deployment like mine.
• Indirect prompt injections: This checks user inputs and tool responses (like files and documents) for hidden instructions. Keeping it on protects your agents when they read external content; turning it off could let a malicious file hijack your model. For an agentic tool like OpenCode that reads files and executes commands, I recommend keeping this enabled, even in Annotate mode.
• Spotlighting (Preview): This tags external content within prompts, allowing the model to differentiate between your actual instructions and random, untrusted data. Think of it as an extra layer of protection against indirect attacks.
• Content harms (Hate, Sexual, Self-harm, Violence): Each of these settings has a severity slider, starting at Medium blocking, which means anything rated medium or higher is blocked. Slide it down to Lowest blocking if you want to reduce false positives like mine. Note that these four categories are mandatory: you can lower the slider, but they cannot be switched off.
• Blocklists: If you want to prevent specific words or phrases, such as profanity or your competitor’s name, this is where to do that. It’s off by default.
• Protected material for code and text: This checks if the model is about to produce known copyrighted material or code. Block stops it; Annotate allows it to pass with a label. If you switch it off, there’s a risk of your app reproducing someone else’s copyrighted work.

Click Next, and Foundry will ask which agents and models this guardrail should apply to. Any previously deployed models in your project will appear here. In my case, that’s gpt-6-astra and gpt-6-sol, so I ticked both.

After clicking Next again, you’ll reach the Review page. Give your guardrail a name (I went for NoGuardrails, because why not?) and choose a streaming mode. You have two options:

• Default: Foundry buffers the response, running it through the guardrail before sending it out. Nothing reaches the user until it’s been checked. This is the safest choice, but you may notice a bit of lag when streaming.
• Asynchronous: Content streams back as soon as it’s generated, with filters running in the background. This option is quicker, but flagged content might briefly appear before it gets filtered, so your app needs to manage that. Microsoft recommends this for low-latency applications like internal dev tools, which fits my situation perfectly.

Take a moment to review the summary, then click Create. You’re all set!

With my newly created guardrail ready, I submitted the same request again, this time to GPT-6 Sol. And just like that, no more content policy issues. The model started working on scanning my footage and assembling the video.

Sure, Microsoft’s default guardrails provide a decent baseline, but they can be overly sensitive when you’re working on legitimate tasks like video editing, creative projects, or internal tools. Since the defaults can’t be modified, the trick is to create your own: switch Block to Annotate where appropriate, adjust those content harm sliders downwards, select the streaming mode that suits your application, and allocate it to your deployed models. That’s all it takes!

However, a friendly reminder: with great power comes great responsibility. Only loosen the settings that you truly need, keep critical features like indirect prompt injection detection active for any agent reading files or browsing the web, and think carefully before applying this to anything that faces customers. If you wish to make more significant adjustments than the portal permits, such as completely disabling content filters, you’ll need to get Microsoft’s approval first. Happy building!