Manage Secrets in Azure Databricks Using Azure Key Vault

 To manage credentials Azure Databricks offers Secret Management. Secret Management allows users to share credentials in a secure mechanism. Currently Azure Databricks offers two types of Secret Scopes:

• Azure Key Vault-backed: To reference secrets stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault. Azure Key Vault-backed secrets are only supported for Azure Databricks Premium Plan.
• Databricks-backed: A Databricks-backed scope is stored in (backed by) an Azure Databricks database. You create a Databricks-backed secret scope using the Databricks CLI (version 0.7.1 and above).

Creating Azure Key Vault

Open a Web Browser. I am using Chrome.

Enter the URL https://portal.azure.com and hit enter.

Sign in into your Azure Account.

After successfully logging to Azure Portal, you should see the following screen.

Click on "All Services" on the top left corner.

Search for "Azure Key Vault" in the "All Services" search text box.

Click on "Key vaults". It will open the blade for "Key vaults".

Click on "Add". It will open a new blade for creating a key vault "Create key vault".

Enter all the information and click the "Create" button. Once the resource is created, refresh the screen and it will show the new "key vault" which we created.

Click on the "key vault" name.

Scroll down and click on the "Properties".

Save the following information for the "key vault" created. We would be using these properties when we connect to the "key Vault" from "databricks"

• DNS Name
• Resource ID

Creating Secret in Azure Key Vault

Click on "Secrets" on the left-hand side.

Click on "Generate/Import". We will be creating a secret for the "access key" for the "Azure Blob Storage".

Enter the required information for creating the "secret".

After entering all the information click on the "Create" button.

Note down the "Name" of the secret.

Creating Azure Key Vault Secret Scope in Databricks

Open a Web Browser. I am using Chrome.

Enter the URL https://portal.azure.com and hit enter.

 

Sign in into your Azure Account.

Open the Azure Databricks workspace created as part of the Azure Databricks Workspace mentioned in the Requirements section at the top of the article.

Click on Launch Workspace to open Azure Databricks.

Copy the "URL" from the browser window.

Build the "URL" for creating the secret scope. https://<Databricks_url>#secrets/createScope.

Enter all the required information:

• Scope Name.
• DNS Name (this is the "DNS name" which we saved when we created the "Azure Key Vault").
• Resource ID (this is the "Resource ID" which we saved when we created the "Azure Key Vault").

Click the "Create" button.

"Databricks" is now connected with "Azure Key Vault".

Using Azure Key Vault Secret Scope and Secret in Azure Databricks Notebook

Open a Web Browser. I am using Chrome.

Enter the URL https://portal.azure.com and hit enter.

Sign in into your Azure Account.

Open the Azure Databricks workspace created as part of the "Azure Databricks Workspace" mentioned in the Requirements section at the top of the article.

Click on "Launch Workspace" to open the "Azure Databricks".

In the left pane, click Workspace. From the Workspace drop-down, click Create, and then click Notebook.

In the Create Notebook dialog box, enter a name, select Python as the language.

Enter the following code in the Notebook

dbutils.secrets.get(scope = "azurekeyvault_secret_scope", key = "BlobStorageAccessKey") 

#azurekeyvault_secret_scope --> Azure Key Vault based scope which we created in Databricks 
#BlobStorageAccessKey --> Secret name which we created in Azure Key Vault 

When you run the above command, it should show [REDACTED] which confirms that the secret was used from the Azure Key Vault secrets.

In the same notebook we are going to add another command section and use Scala as the language.

%scala 
val blob_storage_account_access_key = dbutils.secrets.get(scope = "azurekeyvault_secret_scope", key = "BlobStorageAccessKey") 

//azurekeyvault_secret_scope --> Azure Key Vault based scope which we created in Databricks 
//BlobStorageAccessKey --> Secret name which we created in Azure Key Vault 

When you run the above command, it should show [REDACTED] which confirms that the secret was used from the Azure Key Vault secrets.

References

1. https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks
2. https://azure.microsoft.com/en-us/pricing/details/key-vault/
3. https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks
4. https://docs.azuredatabricks.net/user-guide/secrets/index.html#secrets-user-guide
5. https://docs.azuredatabricks.net/user-guide/secrets/secret-scopes.html
6. https://docs.azuredatabricks.net/user-guide/secrets/secret-scopes.html#create-an-azure-key-vault-backed-secret-scope

Analytics Solution with Azure Databricks

 

Introduction

Databricks is the unified analytics solution powered by Apache Spark, which simplifies data science with a powerful, collaborative, and fully managed machine learning platform. The major analytics solution consists of the following:

Collaborative data science: Simplify and accelerate data science by providing a collaborative environment for data science and machine learning models.

Reliable data engineering: Large-scale data processing for batch and streaming workloads.

Production machine learning: Standardize machine learning life-cycles from experimentation to production.

In this guide, you will learn how to perform machine learning using notebooks in Databricks. The following sections will guide you through five steps to build a machine learning model with Databricks.

Step One: Login to Databricks

The first step is to go to this link and click Try Databricks on the top right corner of the page.

Once you provide the details, it will take you to the following page.

You can select cloud platforms like Azure or AWS. This guide will use the community edition of Databricks. Click on the Get Started tab under the Community Edition option and complete the signup procedures. This will get your account ready, after which you can login into the account with the login credentials.

Step Two: Importing Data

After logging into your account, you will see the following Databricks page.

Click on Import and Explore Data to import the data. You will be uploading the data file Data_db.csv from the local system, and once it is successful, the following output will be displayed.

In the above output, you can see the path of the file, "File uploaded to /FileStore/tables/Data_db.csv". You will use this link later.

Data

This guide will use a fictitious dataset of loan applicants containing 3000 observations and seven variables, as described below:

Income: Annual income of the applicant (in USD)

Loan_amount: Loan amount (in USD) for which the application was submitted

Credit_rating: Whether the applicant's credit score is good (1) or not (0)

Loan_approval: Whether the loan application was approved (1) or not (0). This is the target variable.

Age: The applicant's age in years

Outstanding_debt: The current outstanding debt (in USD) of the applicant from the previous loans.

Interest_rate: The rate of interest being charged by the bank to the applicant.

Step Three: Create Cluster

Again, go back to the Databricks workspace and click on New Cluster under Common Tasks.

This will open the window where you can name the cluster and keep the remaining options to default. Click Create Cluster to create the cluster named "azuredbcluster2".

Once the cluster is created, the following window is displayed. The green circle ahead of the cluster name indicates that the cluster has been successfully created.

Step Four: Launch Notebook

Once again, go back to the welcome page and click on New Notebook under Common Tasks.

The following pop-up will open, and you can fill the preferred input. In this case, you will name the notebook "Databricks ML".

Once the notebook is created, it will display the following output.

Step Five: Build Machine Learning Model

The first step is to load the data, which can be done using the code below.

1
2
3
import pandas as pd
df = pd.read_csv("/dbfs/FileStore/tables/Data_db.csv", header='infer') 
df.head(5)

python

The notebook view is displayed below along with the output.

The next step is to load the other required libraries and modules.

1
2
3
4
5
6
7
8
# Import other required libraries
import sklearn
import numpy as np

# Import necessary modules
from sklearn.linear_model import LogisticRegression 
from sklearn.model_selection import train_test_split  
from sklearn.metrics import confusion_matrix, classification_report

python

Also, create the train and test arrays required to build and evaluate the machine learning model.

1
2
3
4
5
6
7
# Create arrays for the features and the response variable
y = df['Loan_approval'].values
X = df.drop('Loan_approval', axis=1).values 

# Create training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.4, random_state=42) 
X_train.shape, X_test.shape

python

The notebook view is displayed below along with the output.

From the above output, you can see that there are 1800 observations in train data and 1200 observations in test data.

Next, create the logistic regression classifier, logreg, and fit the classifier on the train data.

1
2
3
4
5
# Create the classifier: logreg
logreg = LogisticRegression()

# Fit the classifier to the training data
logreg.fit(X_train, y_train)

python

The above code will generate the output displayed in the notebook view below.

You have trained the model and the next step is to predict on test data and print the evaluation metrics. This is done with the code below.

1
2
3
4
5
6
# Predict the labels of the test set: y_pred
y_pred = logreg.predict(X_test)

# Compute and print the confusion matrix and classification report
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

python

The above code will generate the output displayed in the notebook view below.

The output above shows that the model accuracy is 67.25% while the sensitivity is 79%. The model can be further improved by doing cross-validation, features analysis, and feature engineering and, of course, by trying out more advanced machine learning algorithms. To learn more about these and other techniques with Python, please refer to the links given at the end of the guide.