> For the complete documentation index, see [llms.txt](https://azure.kasinathanramesh.vip/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://azure.kasinathanramesh.vip/azure-data-lake-storage-gen2/deployment-steps.md).

# Deployment Steps

## Create an Azure storage account <a href="#create-an-azure-storage-account" id="create-an-azure-storage-account"></a>

#### How to create an ADLS Gen 2 storage account:

Open your Azure portal, find and go to the Storage accounts, and click on the + Create button to create a new storage account.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FTvVgPGrg8NfytpjvB33X%2Fimage.png?alt=media&amp;token=60e12ba3-ee94-40fe-aeb4-aee0e0b255cd" alt=""><figcaption></figcaption></figure>

In the basics tab, select your subscription, select your resource group, name your ADLS Gen 2 storage account, select your region, then select performance as Premium, select the account type as Block blobs, and then go to the advanced tab.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FsNz52M6tIGJXByqribpT%2Fimage.png?alt=media&amp;token=3b619b2b-7838-46d0-bd46-ba5c1867906a" alt=""><figcaption></figcaption></figure>

In the advanced tab enable the Hierarchical namespace, then click on Review + create and then click on create.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FHcee8LFTC4r5cIUASy3g%2Fimage.png?alt=media&amp;token=de882dd2-9a61-4dd6-b1a2-d013534a6bf2" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FZpKndmwwdwLHGneI4mMa%2Fimage.png?alt=media&amp;token=7ae444e2-ad08-431e-8a26-b85e9cf227a0" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2F3rx89FZ14b6foS4fBPfo%2Fimage.png?alt=media&amp;token=7623aecf-abe5-4045-b043-28d2a03ebe6e" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FCYT0brEFQgdwd4TSoCIC%2Fimage.png?alt=media&amp;token=e890e067-b7d5-4b31-94aa-142fe3a73548" alt=""><figcaption></figcaption></figure>

Once our ADLS Gen2 storage account is created, go to the ADLS Gen2 storage, click on containers/folders under the data storage tab, then click on + Create to create a container/folder.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FyLJJk9GA51SVYVbSBmvT%2Fimage.png?alt=media&amp;token=e79664db-affb-45c3-b98b-f49dad91dddc" alt=""><figcaption></figcaption></figure>

## &#x20;<a href="#tutorial-connect-to-azure-data-lake-storage-gen2" id="tutorial-connect-to-azure-data-lake-storage-gen2"></a>

## &#x20;<a href="#tutorial-connect-to-azure-data-lake-storage-gen2" id="tutorial-connect-to-azure-data-lake-storage-gen2"></a>

## Connect to Azure Data Lake Storage Gen2 <a href="#tutorial-connect-to-azure-data-lake-storage-gen2" id="tutorial-connect-to-azure-data-lake-storage-gen2"></a>

1. [Step 1: Create a Microsoft Entra ID service principal](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-1-create-a-microsoft-entra-id-service-principal)
2. [Step 2: Create a client secret for your service principal](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-2-create-a-client-secret-for-your-service-principal)
3. [Step 3: Grant the service principal access to Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-3-grant-the-service-principal-access-to-azure-data-lake-storage-gen2)
4. [Step 4: Add the client secret to Azure Key Vault](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-4-add-the-client-secret-to-azure-key-vault)
5. [Step 5: Create Azure Key Vault-backed secret scope in your Azure Databricks workspace](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-5-create-azure-key-vault-backed-secret-scope-in-your-azure-databricks-workspace)
6. [Step 6: Connect to Azure Data Lake Storage Gen2 using python](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--step-6-connect-to-azure-data-lake-storage-gen2-using-python)
7. [Grant your Azure Databricks workspace access to Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#--grant-your-azure-databricks-workspace-access-to-azure-data-lake-storage-gen2)
8. [Troubleshooting](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#troubleshooting)
9. [Resources](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#resources)

Connect from Azure Databricks to Azure Data Lake Storage Gen2 using OAuth 2.0 with a Microsoft Entra ID service principal.

### Step 1: Create a Microsoft Entra ID service principal <a href="#step-1-create-a-microsoft-entra-id-service-principal" id="step-1-create-a-microsoft-entra-id-service-principal"></a>

To use service principals to connect to Azure Data Lake Storage Gen2, an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application.

To create a Microsoft Entra ID service principal, follow these instructions:

Sign in to the [Azure portal](https://portal.azure.com/)

If you have access to multiple tenants, subscriptions, or directories, click the **Directories + subscriptions** (directory with filter) icon in the top menu to switch to the directory in which you want to provision the service principal.

Search for and select \<Microsoft Entra ID.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FWL3O4kzyFfzwvl1BRHTw%2Fimage.png?alt=media&amp;token=203579ed-a133-48bc-b046-b5a3b0f86c20" alt=""><figcaption></figcaption></figure>

In Manage, click App registrations > New registration.

In the **Supported account types** section, select **Accounts in this organizational directory only (Single tenant)**.

Click **Register**.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FG1cG32aEuN2TouT6CrDM%2Fimage.png?alt=media&amp;token=3ac63065-5ded-4f47-9216-44d0614cd5d9" alt=""><figcaption></figcaption></figure>

### Step 2: Create a client secret for your service principal <a href="#step-2-create-a-client-secret-for-your-service-principal" id="step-2-create-a-client-secret-for-your-service-principal"></a>

In **Manage**, click **Certificates & secrets**.

On the **Client secrets** tab, click **New client secret**.

In the **Add a client secret** pane, for **Description**, enter a description for the client secret.

For **Expires**, select an expiry time period for the client secret, and then click **Add**.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2F4QELgvJxNAJFWEjpcgU4%2Fimage.png?alt=media&amp;token=58be2eb0-e39d-4334-b9a3-52cc4f567259" alt=""><figcaption></figcaption></figure>

Copy and store the client secret’s **Value** in a secure place, as this client secret is the password for your application.

Secret ID: 9660b019-c99c-40b9-ac60-e3bfb24015c6

Value: jQA8Q\~25zzIGaAX7svVPC9wjKnp-ffgxX3LjNaKy

On the application page’s **Overview** page, in the **Essentials** section, copy the following values:

* **Application (client) ID**
* **Directory (tenant) ID**

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2F3cGt7tSbXzefyHTy6wwM%2Fimage.png?alt=media&amp;token=6193fe9a-51bc-49e4-a8d9-d539b1468550" alt=""><figcaption></figcaption></figure>

### Step 3: Grant the service principal access to Azure Data Lake Storage Gen2 <a href="#step-3-grant-the-service-principal-access-to-azure-data-lake-storage-gen2" id="step-3-grant-the-service-principal-access-to-azure-data-lake-storage-gen2"></a>

You grant access to storage resources by assigning roles to your service principal. In this tutorial, you assign the **Storage Blob Data Contributor** to the service principal on your Azure Data Lake Storage Gen2 account. You may need to assign other roles depending on specific requirements.

1. In the Azure portal, go to the **Storage accounts** service.
2. Select an Azure storage account to use.
3. Click **Access Control (IAM)**.
4. Click **+ Add** and select **Add role assignment** from the dropdown menu.
5. Set the **Select** field to the Microsoft Entra ID application name that you created in step 1 and set **Role** to **Storage Blob Data Contributor**.
6. Click **Save**.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FPOCfegkbH7klKEcQTVqf%2Fimage.png?alt=media&amp;token=3220316a-f606-45d8-89c3-ff5ae1fea464" alt=""><figcaption></figcaption></figure>

### Step 4: Add the client secret to Azure Key Vault <a href="#step-4-add-the-client-secret-to-azure-key-vault" id="step-4-add-the-client-secret-to-azure-key-vault"></a>

ou can store the client secret from step 1 in Azure Key Vault.

1. In the Azure portal, go to the **Key vault** service.
2. Select an Azure Key Vault to use.
3. On the Key Vault settings pages, select **Secrets**.
4. Click on **+ Generate/Import**.
5. In **Upload options**, select **Manual**.
6. For **Name**, enter a name for the secret. The secret name must be unique within a Key Vault.
7. For **Value**, paste the Client Secret that you stored in Step 1.
8. Click **Create**.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2FTqiUSf3WlylV2aZSUo6I%2Fimage.png?alt=media&amp;token=0c9b775a-c27b-4609-bd9c-04f45559c8c0" alt=""><figcaption></figcaption></figure>

### Step 5: Create Azure Key Vault-backed secret scope in your Azure Databricks workspace <a href="#step-5-create-azure-key-vault-backed-secret-scope-in-your-azure-databricks-workspace" id="step-5-create-azure-key-vault-backed-secret-scope-in-your-azure-databricks-workspace"></a>

To reference the client secret stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault in Azure Databricks.

Go to `https://<databricks-instance>#secrets/createScope`. This URL is case sensitive; scope in `createScope` must be uppercase.

<figure><img src="https://3715319556-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F298nB6EDtaq3VeKvolh7%2Fuploads%2Fpzb1FHyigHRDNl6McpMj%2Fimage.png?alt=media&amp;token=23d13b85-0757-46a6-b02d-54aee9f4254e" alt=""><figcaption></figcaption></figure>

{% embed url="<https://learn.microsoft.com/en-us/azure/private-link/tutorial-private-endpoint-storage-portal?tabs=dynamic-ip>" %}