Enabling the Compliance Security Profile (CSP) for HIPAA on Azure Databricks
Microsoft Architect’s: Aladdin Alchalabi aalchalabiโ, Kiran Raja, Anishek Kamal anishekkamalโย and Rafia Aqil Rafia_Aqilโย
Azure Databricks has implemented controls to meet the unique compliance needs of highly regulated industries. The Compliance Security Profile (CSP) is a joint effort between Microsoft and Databricks for Azure Databricks workspaces. The value proposition of the Compliance Security Profile is that it provides Customers with significantly more hardening and security features.ย ย
- Mandatory Deadline:โฏThe Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP-regulated data on Azure Databricks byโฏSeptember 1, 2026.ย
We check and enforce these requirements on new workspaces today, with enforcement on existing workspaces expected in the future; if prerequisites are missing, clusters may fail to start.ย
| Prerequisiteย | Requirementย |
| Costsย | CSP adds a 10% cost to Azure Databricks product spend within each workspace where CSP is enabled.ย **Review with your account team for any grace period during which the Enhanced Security & Compliance (ESC) add-on is available at no charge. After the grace period ends, a 10% DBU upcharge applies.ย |
| Enhanced Security & Compliance add-onย | For existing workspaces: From the Azure portal, click Settings > Security & compliance in an existing Azure Databricks workspace: ย ย |
| Azure VNet encryptionย | Azure Virtual Network encryption must be enabled on the Azure Databricks workspace VNet. In a hub-and-spoke design, this is typically the Databricks spoke VNet.ย |
| Supported VM instance typesย | Use a VM series that supports VNet encryption and verify compatibility before enabling the profile. This does not apply to Serverless Computeย |
NOTE: ย
- You can enable the profile when you create a workspace or on an existing workspace through the Azure portal, Azure CLI, PowerShell, an ARM template, or Terraform.ย ย
- Only the Public Preview, Private Preview, and Beta features listed in this section are supported for workspaces with the compliance security profile enabled:ย Compliance security profile – Azure Databricks | Microsoft Learnย
The steps below represent a validated implementation pattern. The exact network design can vary by environment, but apply the same prerequisite, isolation, and end-to-end validation principles.ย
| Validated implementation stepย | Recommended approach and expected outcomeย |
| Isolated sandbox workspaceย | Enable CSP first in a representative non-production workspace. This avoids irreversible changes to DEV or production while you validate the network topology, dependencies, VM compatibility, and operational behaviour.ย |
| Enable CSP and select HIPAAย | Enable the Compliance Security Profile and select HIPAA under Settings > Security & compliance before processing PHI after September 1, 2026.ย |
| Enable VNet encryptionย | Enable VNet encryption. **Please review Azure Virtual Network encryption limitations:ย What is Azure Virtual Network encryption? – Azure Virtual Network | Microsoft Learnย |
| Start a classic clusterย | Confirm a classic cluster starts successfully after you apply the CSP and VNet encryption prerequisites. This validates that the selected compute path and VM types remain operational.ย |
| Validate storage connectivity | Confirm storage connectivity continues to work.ย |
| Confirm rollout readinessย | Proceed to DEV and production only after you validate the complete private connectivity path, cluster startup, storage access, DNS resolution, data pipelines, and end-to-end performance.ย |
Enabling the compliance security profile, or adding a compliance standard, is intended to be a permanent change. You cannot remove the profile or an individual standard from a workspace that has ever processed regulated data; to revert, you must delete the workspace and create a new one. Validate the configuration in an isolated, representative non-production workspace before enabling DEV or production.ย
Inventory and Assessmentย
- Identify Regulated Workspaces:โฏCatalogue all existing Azure Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.ย
- Review Data Pipelines:โฏMap out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.ย
Verify Prerequisites Before Rollout:โฏConfirm that the selected VM instance types support VNet encryption and that all required CSP and networking settings are in place, because missing prerequisites can prevent clusters from starting.ย
- Enablement Method:โฏChoose the appropriate tooling, Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform, to ensure consistency and automation.ย
Keep sensitive data out of customer-defined fieldsย
You are solely responsible for ensuring that PHI or other sensitive information is never entered into customer-defined input fields. These include workspace names, compute and resource names, tags, job names, job run names, network names, credential names, storage account names, and Git repository IDs or URLs, all of which may be stored, processed, or accessed outside the compliance boundary.ย
What Changes After Enabling Compliance Security Profileย ย
On CSP-enabled workspaces, Partner-powered AI features are disabled by default, and some assistive features, such as Genie Code, are also disabled; a workspace admin can re-enable them if required. In addition, only the specific preview features listed in the compliance security profile documentation are supported. No other Public Preview, Private Preview, or Beta feature may be used to process regulated data.ย
ย
Share this content:
Discover more from Qureshi
Subscribe to get the latest posts sent to your email.
