Governing Log Analytics retention at scale

A Log Analytics workspace stores data in tables, and you can configure how long this data is retained at two levels:

• Workspace level: This sets a default interactive (Analytics) retention for the entire workspace, ranging from 30 to 730 days.
• Table level: Each table has its specific Analytics retention period and a separate total retention period (which includes both Analytics and long-term/archive). By default, a table’s Analytics retention is set to “Workspace default”, meaning it inherits the retention setting from the workspace.

Here are two key points to remember:

• Understanding Analytics vs. long-term (archive) retention: The Analytics retention is the active, fully queryable period. After this, data can be stored in a cheaper long-term retention option for compliance purposes and can be accessed when required. For more details, check out how to manage data retention in a Log Analytics workspace.
• Individual table configuration: Instead of setting retention at the highest level (workspace), you can set it at the table level for more tailored control.

Let’s look at a common scenario.

A customer wants to keep their data for 2 years. The usual response: set the workspace retention to 730 days.

The issue here is that since each table’s Analytics retention is set to “Workspace default”, it will unintentionally apply a retention period of two years to all tables. Since Analytics retention is usually more costly, this choice can lead to higher storage expenses for data that isn’t accessed interactively after the initial weeks.

A more economical approach often involves:

1. Setting a lower default retention for the workspace (e.g., 30 or 90 days, depending on whether you have Sentinel and/or Application Insights active).
2. Adjusting each table’s Analytics retention according to its actual needs, which often falls between 30 and 90 days.
3. Implementing long-term (non-interactive) retention at the table level for data that genuinely needs to be stored longer (e.g., 2 years) for compliance, at a significantly lower cost than Analytics. This can go up to a maximum of 12 years.

This approach keeps the same goal of retaining data for 2 years but ensures that only the data actually needed for active queries remains in the more expensive tier.

Setting this up once in the portal is quite simple. The real value lies in making your settings consistent, scalable, and self-sustaining—and achieving this is quite feasible with the right strategy:

• A single workspace can encompass hundreds or even thousands of tables, so having a repeatable and automated method will prove to be a wise investment.
• By having a central definition of your desired retention model, you can apply it uniformly across multiple workspaces and subscriptions.
• Incorporating drift detection means you will always be aware of any retention changes for tables and can correct them automatically.

In essence, managing retention is an excellent candidate for policy-driven automation. To facilitate this, I have developed a reusable solution that’s open-sourced for public use.

Repository: claestom/law-retention-guardrails

The solution operates using Azure Policy from start to finish. It comes with two custom policy definitions that are grouped into one initiative, employing the DeployIfNotExists effect:

Because it uses DeployIfNotExists, the policy performs both tasks simultaneously:

• Audit: You can monitor compliance via Azure Policy → Compliance. This highlights any workspace and table whose retention settings deviate from your specified target across the board.
• Remediate: Newly created and updated resources are automatically configured, and a remediation task helps bring existing workspaces and tables into compliance. The managed identity for the assignment (given Log Analytics Contributor permissions) executes the required changes.

One thing to note: enabling Microsoft Sentinel in a workspace, or using workspace-based Application Insights, grants 90 days of interactive (analytics) retention at no extra cost. If you set a target of just 30 days, you risk losing this benefit.

Application Insights is managed for you. The initiative includes a specific third policy aimed precisely at the 11 workspace-based App Insights tables (such as AppRequests, AppDependencies, AppExceptions, AppTraces, etc.). The overarching table policy excludes these tables, preventing any overlap; thus, a single assignment manages everything, while App Insights maintains its own settings (appInsightsRetentionInDays / appInsightsTotalRetentionInDays, which defaults to 90 / 90) while your other tables adhere to the standard.

Sentinel shows separate scope. The free 90 days for Sentinel is applied at the workspace level, which is not covered by the table exclusions. Since Sentinel usually operates in a dedicated workspace or resource group, ensure the 90-day values are assigned there and exclude that scope from the standard settings with -NotScopes. Keep any additional assignments distinct; having two DeployIfNotExists assignments matching the same resource could lead to conflicts.

To create policy and role assignments at the target scope (e.g. Owner rights) is important. You can choose from two paths.

The repository provides a Deploy to Azure button that works through a subscription-scoped ARM template. This creates the three definitions, the initiative, the assignment (complete with a managed identity), and the Log Analytics Contributor role assignment — you simply enter the retention values in the portal form. Optionally, you can narrow the assignment to a single resource group right in the form.

git clone https://github.com/claestom/law-retention-guardrails.git
cd law-retention-guardrails

./deploy.ps1 -SubscriptionId  `
  -WorkspaceRetentionInDays 30 `
  -TableRetentionInDays 30 `
  -TableTotalRetentionInDays 730 `
  -AppInsightsRetentionInDays 90 `
  -AppInsightsTotalRetentionInDays 90

That single command sets up the definitions and initiative, assigns it with a system-assigned managed identity, grants the identity Log Analytics Contributor, and starts a remediation task to align existing resources with the new settings.

You can specify the scope as broadly or narrowly as needed:

# A management group
./deploy.ps1 -ManagementGroupId 

# A single resource group
./deploy.ps1 -SubscriptionId  -ResourceGroupName rg-monitoring

# A single Log Analytics workspace
./deploy.ps1 -SubscriptionId  `
  -Scope /subscriptions//resourceGroups//providers/Microsoft.OperationalInsights/workspaces/

Prefer using the portal? You can paste each definition’s azurepolicy.portal.json into Policy → Definitions → + Policy definition, and then assign it at your desired scope using a managed identity and a remediation task.

Azure accepts these total retention values: from 4 to 730 days, and for anything over two years, only full years are accepted: 1095, 1460, 1826, 2191, 2556, 2922, 3288, 3653, 4018, 4383. The solution ensures this is checked beforehand to prevent confusion, providing a clear message instead of generating errors for each table. (Note: Basic/Auxiliary plan tables have a fixed analytics retention and will always return non-compliant—be sure to exempt them or treat them as anomalies.)

By configuring retention per table, keeping the default workspace retention low, refining interactive retention, and directing long-term data into more affordable long-term retention, you can significantly lower Azure Monitor expenses while still fulfilling compliance obligations. With law-retention-guardrails, this model can be confidently enforced and audited throughout your entire environment using only Azure Policy: it not only indicates where you stand but also corrects deviations without the need for an Automation Account.

Your feedback and contributions to the repository are welcome.

Thank you!