Enabling the Compliance Security Profile (CSP) for HIPAA on Azure Databricks
Microsoft Architect’s: Aladdin Alchalabi aalchalabi, Kiran Raja, Anishek Kamal anishekkamal and Rafia Aqil Rafia_Aqil
Azure Databricks has implemented controls to meet the unique compliance needs of highly regulated industries. The Compliance Security Profile (CSP) is a joint effort between Microsoft and Databricks for Azure Databricks workspaces. The value proposition of the Compliance Security Profile is that it provides Customers with significantly more hardening and security features.
- Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP-regulated data on Azure Databricks by September 1, 2026.
We check and enforce these requirements on new workspaces today, with enforcement on existing workspaces expected in the future; if prerequisites are missing, clusters may fail to start.
| Prerequisite | Requirement |
| Costs | CSP adds a 10% cost to Azure Databricks product spend within each workspace where CSP is enabled. **Review with your account team for any grace period during which the Enhanced Security & Compliance (ESC) add-on is available at no charge. After the grace period ends, a 10% DBU upcharge applies. |
| Enhanced Security & Compliance add-on | For existing workspaces: From the Azure portal, click Settings > Security & compliance in an existing Azure Databricks workspace:
|
| Azure VNet encryption | Azure Virtual Network encryption must be enabled on the Azure Databricks workspace VNet. In a hub-and-spoke design, this is typically the Databricks spoke VNet. |
| Supported VM instance types | Use a VM series that supports VNet encryption and verify compatibility before enabling the profile. This does not apply to Serverless Compute |
NOTE:
- You can enable the profile when you create a workspace or on an existing workspace through the Azure portal, Azure CLI, PowerShell, an ARM template, or Terraform.
- Only the Public Preview, Private Preview, and Beta features listed in this section are supported for workspaces with the compliance security profile enabled: Compliance security profile – Azure Databricks | Microsoft Learn
The steps below represent a validated implementation pattern. The exact network design can vary by environment, but apply the same prerequisite, isolation, and end-to-end validation principles.
| Validated implementation step | Recommended approach and expected outcome |
| Isolated sandbox workspace | Enable CSP first in a representative non-production workspace. This avoids irreversible changes to DEV or production while you validate the network topology, dependencies, VM compatibility, and operational behaviour. |
| Enable CSP and select HIPAA | Enable the Compliance Security Profile and select HIPAA under Settings > Security & compliance before processing PHI after September 1, 2026. |
| Enable VNet encryption | Enable VNet encryption. **Please review Azure Virtual Network encryption limitations: What is Azure Virtual Network encryption? – Azure Virtual Network | Microsoft Learn |
| Start a classic cluster | Confirm a classic cluster starts successfully after you apply the CSP and VNet encryption prerequisites. This validates that the selected compute path and VM types remain operational. |
| Validate storage connectivity | Confirm storage connectivity continues to work. |
| Confirm rollout readiness | Proceed to DEV and production only after you validate the complete private connectivity path, cluster startup, storage access, DNS resolution, data pipelines, and end-to-end performance. |
Enabling the compliance security profile, or adding a compliance standard, is intended to be a permanent change. You cannot remove the profile or an individual standard from a workspace that has ever processed regulated data; to revert, you must delete the workspace and create a new one. Validate the configuration in an isolated, representative non-production workspace before enabling DEV or production.
Inventory and Assessment
- Identify Regulated Workspaces: Catalogue all existing Azure Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.
- Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.
Verify Prerequisites Before Rollout: Confirm that the selected VM instance types support VNet encryption and that all required CSP and networking settings are in place, because missing prerequisites can prevent clusters from starting.
- Enablement Method: Choose the appropriate tooling, Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform, to ensure consistency and automation.
Keep sensitive data out of customer-defined fields
You are solely responsible for ensuring that PHI or other sensitive information is never entered into customer-defined input fields. These include workspace names, compute and resource names, tags, job names, job run names, network names, credential names, storage account names, and Git repository IDs or URLs, all of which may be stored, processed, or accessed outside the compliance boundary.
What Changes After Enabling Compliance Security Profile
On CSP-enabled workspaces, Partner-powered AI features are disabled by default, and some assistive features, such as Genie Code, are also disabled; a workspace admin can re-enable them if required. In addition, only the specific preview features listed in the compliance security profile documentation are supported. No other Public Preview, Private Preview, or Beta feature may be used to process regulated data.
Share this content:
Discover more from Qureshi
Subscribe to get the latest posts sent to your email.
