Azure Machine Learning - Writing Python Script In Notebook

In this article, we’ll learn on step-by-step process to write python script in Notebook offered in the Azure Machine Learning Studio. This will come handy when we want to work on Machine Learning Projects. This article is a part of the Azure Machine Learning Series.

1. Azure Machine Learning - Create Workspace for Machine Learning  
2. Azure Machine Learning – Create Compute Instance and Compute Cluster  
3. Azure Machine Learning - Writing Python Script in Notebook  
4. Azure Machine Learning - Model Training 

Microsoft AI

Microsoft AI is a powerful framework that enables organizations, researchers, and non-profits to use AI technologies with its powerful framework which offers services and features across domains of Machine Learning, Robotics, Data Science, IoT, and many more.  Learn more about Microsoft AI from this article.

Azure Machine Learning

The Azure Machine Learning enriches and consolidates the functionalities to support model training and deployment which transitions from Machine Learning Studio. It provides tools for Machine Learning works for all skill levels, provides an open and interoperable framework with support to different languages, and enables robust end-to-end MLOps. It also supports Automated Machine Learning. Read this article Auto ML to learn more about it.

So, where and how do we start if we want to create and deploy a Machine Learning project? Azure Machine Learning provides all the tools through its portal to create the resources and set up the infrastructure that is needed for any kind of machine learning works.

Pre-requisite 

Before we start with the tutorial of this article, you first need to create Machine Learning Workspace in Azure and create compute instance along with compute cluster. Follow up the Azure Machine Learning - Create Workspace for Machine Learning  and Azure Machine Learning – Create Compute Instance and Compute Cluster respectively.

Now, as the above resources have been created, we can now focus on writing the Python script on Notebook in Azure.

Step 1

Open the Azure Machine Learning Studio.

Select the Notebooks from the left-hand side. You’ll see the user listed there.

Step 2

Next, under the Files, Click on the + Button and Select Create new folder.

Step 3

You’ll be asked to name the new folder.

I’m naming learn-ml. You can follow it too. Next, Click on Create

Step 4

Now, under learn-ml, you’ll see three … on the right-hand side. Click it and you’ll be listed with the options.

Select, Create new folder again.

Step 5

We name this folder, src.

Click on Create and the new folder src will be created inside learn-ml.

Step 6

Now, on the right-hand side, click the … option on src and select Create new file.

Step 7

Now, we’ll be provided with the details to fill up for the file name and file type.

The default selection will be .ipynb. This will be for juypter notebook creation. We are instead writing a python script file instead.

For this, select .py under the file type.

Name your file as trial.py and then click on Create.

Step 8

Notification pops up with the confirmation of the file creation.

Step 9

Now, visit the file. You’ll see the empty file as of now.

You can see, the file is running under the compute instance, ojashshrestha11 that I created.

You’ll sometimes be asked to authenticate the Azure SDK. Click on the Authenticate button and once done you’ll be shown with the confirmation.

Writing Python Script

Step 10

Now, let us write some code on the trial.py file.

# src/trial.py
print("This is our first experience with ML Notebook in Azure")

Here, I’ve written simply a code to print the text, “This is our first experience with ML Notebook in Azure.”

Step 11

Now, to run this, click on the Next icon on top which will save and run the code.

Step 12

We are now directed to the terminal to print the output of the file.

You can see this is running on the compute instance we had created from the earlier article.

Step 13

Now, let us close this terminal. Click on View Active Sessions to click the active sessions.

Simply Click on the X button and you’ll be asked for confirmation to Terminate the Process.

Step 14

Now, under the learn-ml folder, let's create another file and name it run-trial.py

As the file is successfully created, we are updated with a notification.

Writing Control Script

Step 15

Now, Copy & paste the following code to the file.

# get-started/run-trial.py
from azureml.core import Workspace, Experiment, Environment, ScriptRunConfig

ws = Workspace.from_config()
experiment = Experiment(workspace=ws, name='day1-experiment-try1')

config = ScriptRunConfig(source_directory='./src', script='trial.py', compute_target='cpu-cluster')

run = experiment.submit(config)
aml_url = run.get_portal_url()
print(aml_url)

Here, we are creating the experiment day1-experiment-try1. This code will help run the script trial.py under the src folder using the cpu-cluster.

Here, I’ve named my cpu-cluster is named as ojashshrestha11 so, I’ll name the compute_target as that.

Step 16

Now, as we click on the save and run button, the terminal will open executing the script.

You can see the link of the experiment in the workspace in the terminal. Click on it.

Step 17

You can see the status of the experiment is Preparing.

As it is successfully run, click on refresh and we can see the status Completed. Here, the initial run can take anywhere between 5 to 10 minutes as the docker image is built in the cloud with compute cluster being resized and downloaded to the compute. Later runs will be run within 30 seconds.

Step 18

Now, Visit the Experiments and under the Outputs + Logs we can find azureml-logs. Open the file 70_driver_log.txt

 

This file consists of all the logs of the process. It is extremely helpful when we need to debug remote runs in the cloud as we go deeper.

Step 19

On line 16, we can see your printed output, “This is our first experience with ML Notebook in Azure”.

This shows that our output can also be viewed on the log which was successfully run. During error cases, we can find out the reasons here which will make it convenient for us to debug.

Deletion and Removal of Resources 

Step 20

As we are done using the resources, it is essential to shut down the resources and even better deletion of the resources for a lot of these services incur charges as they are running even if we are not using it. Hence, to save from it, it's better to delete all the resources created.

 

To perform this, we have two ways. One is deleting each service in the Azure Machine Learning Studio and another is simply deleting the resource group which consists of the workspace and other resources such as Container Registry, Storage Account, Key Vault and Application Insight which was created in order to access all these ML Services to us.

Visit the Azure Portal and Click on Delete resource group selecting the resource group you set up the workspace.

Conclusion

Thus, in this article, we learned in step-by-step process to run python scripts in Notebook using the Machine Learning Studio in Azure. Then we explored writing a control script to run our trial.py on various compute resources as we choose. We also then learned to explore the log of the processes. Hence, we learned to connect our Machine Learning workspace and write python scripts and even control scripts on it to create experiments and submit our code in Azure.