Deploying An Azure Batch Job From Azure Portal

Why use Azure Batch Service

 

Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. Azure Batch creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes. There is no cluster or job scheduler software to install, manage, or scale. Instead, you use Batch APIs and tools, command-line scripts, or the Azure portal to configure, manage, and monitor your jobs.

 

Developers can use Batch as a platform service to build SaaS applications or client apps where large-scale execution is required. For example, build a service with Batch to run a Monte Carlo risk simulation for a financial services company, or a service to process many images.

 

Deployment options for Azure Batch Service

 

There are two ways that you can go about deploying your batch Jobs. And, I'm going to call it directly versus indirectly.

 

 

You could either be in the portal walking through the GUI and walking through the steps, or you could bring off the Cloud Shell and run things with Azure CLI. Or you could do it indirectly using code in the API. When we deploy Azure Batch, it will generate access keys, which we can use to access the Azure Batch service over an API. We could deploy our code in all of our processes that way. But in this article, we're going to be jumping into the portal and deploying a sample app for the Azure Batch just to see how the batch would actually work in the GUI.

 

Overview of the task at hand

 

You can see the steps below here that we're going to be following. I've already created the resource group and the storage account already because there's really not much to do those. You just create them. So, what we're then going to do next is we're going to be actually creating the batch service. Then we're going to be creating the pool, the job, and the tasks. And once the tasks are created, they will automatically start going through and deploying the tasks to any available VM’s within the pool. So, we'll be able to monitor our progress and then see the results in the Azure storage.

 

 

So, let's jump into the portal and get started. Of course, I am logged into portal.azure.com. I've gone through the multi-factor authentication process, and now I am ready to deploy my Azure batch. To do that, we're just going to click Create a Resource here in the very top left. And I'm subsequently going to search for the batch. And guess what? Batch Service right there-- that's what I'm trying to do. We'll go and click Batch Service. And we'll click Create.

 

 

Now the resource group that I created for this particular demo was called BatchServiceDemo -- pretty straightforward. And it's located in Southeast Asia you can see that when we do that, we get a prompt to fill out the account name, which has to be a globally unique fully qualified domain name (FQDN). And it specifically identifies the region that we're going to be deploying to

 “southeastasia.batch.azure.com”

 

So, I'll just call this batchservicedemo, and click off. Make sure we get the green check here. So, it's going to be, batchservicedemo.southeastasia.batch.azure.com.

 

And if you scroll down, it says you can specify an optional storage account or it's just going to create one if you don't already have one in the existing resource group or region.

 

 

I'll just go ahead and click Select a Storage Account since I already created one called batchservicedemo right here. We'll go and give that a click. And if we scroll on down, we can see all of my selections have been made. I can now click Review and Create.

 

 

Validation has passed, and just remember that you can always download the template for automation. So, if you ever want to deploy this from code later, you do have things like the Azure CLI, PowerShell, and all of the other good stuff available to you. I'll just go and click back to the new Azure Batch account here and click Create.

 

 

And it'll create my new batch service. And this only took 19 seconds to deploy because guess what? We haven't really deployed the virtual machines in the pool yet, right? This is just a shell at this point. So, we'll click Go to Resource, and we'll start building out our pool of virtual machines.

 

 

So, we'll scroll on down just a hair. And under Features, you can see pools. That's where we're going to be deploying our virtual machines.

 

Now as an aside, if you had custom-built applications that are actually going to be handling the processing for you, you would click on Applications and upload it here. And once we've deployed our pool, we're going to be going to jobs, where we're going to create the jobs and the tasks.

 

 

So, let's start off with the pools. And we're going to click Add right here under this so that we can create our first pool.

 

 

The pool ID, I'll just go and give this a name like mypool, something simple enough like that. The display name will be “my pool”. The image type, we're going to pull down our image from the marketplace. But you can see that we can get custom images or even specifics images that are built around graphics and rendering if we're going to be doing other sorts of specific tasks. But generic Windows is fine for my particular demo here. The publisher is going to be Microsoft Windows Server. The offer, of course, is going to be Windows Server.

 

 

That's the only item that we have. And the SKU is basically, which addition do you want to run, I'm going to choose 2019 Data Center Small Disk.

 

Now obviously, at this point, you get into a lot of customization options that you have here, such as data disk. So, we're going to need extra local storage on these VMs that we're going to be able to add our data disks here. Graphics and rendering licensing-- if you actually click this here, you can see we can deploy things like Autodesk or Chaos V-Ray. The beautiful thing about the cloud is you only pay for what you use. This is all billed per hour. It's not billed for the actual license itself, which is really, really great news. But I don't need these for this particular demo. Just know this is available to you if you need to add additional modeling or licensing to your pool.

 

 

Now the node size and scale-- this is where this gets really important. What size VM do we want to be spinning up in our cluster or in our batch. I'm going to choose A2 for right now so that we've got a dedicated two cores and 3 and 1/2 gigs of RAM per machine. But you can see we can scale up pretty high if you wanted to, and there's a whole lot of options available to you too. Keeping it simple, though, I'm just going to go with A2 right now. In the scale, how many machines do we actually want? We can set a fixed number of machines that we want to spin up, or we can even do auto-scale. And we can auto-scale based on a formula where we can choose things like CPU percentage or RAM percentage.

 

 

I'm going to swap it back to fixed really quick and the dedicated nodes. This very first question, is your target dedicated nodes. This is the number of machines that you are guaranteeing is going to be up and running on your hardware. Now the low priority nodes is an interesting item where it could essentially spin up a node that is just going to be running on lower priority or lower preference hardware, and it gets billed at a lower rate. If they do get spun up and start processing tasks, great. You're going to be getting it at a lower cost. But they might not be spun up if other higher priority nodes are being requested at that time.

 

 

Now the other key options that you may want to explore, first of all, are user accounts if you have to run within a certain context of your specific tasks that you're trying to do. Or if you want to scroll on down, you could set certificates if you have some sort of outbound communication taking place.

 

 

Then you have virtual networking, where you can configure inbound NATs or if you want to connect it to a specific VNet and subnet from there too. I don't need to worry about any of these particular items right now, so I'm going to click OK. And that's going to start spinning up my VMs in my pool.

 

 

You can see the allocation state right now is in resizing. It's going to be there for a few minutes while it actually spins up the virtual machines and sizes them correctly.

 

 

So, while this is going on, I'm going to hop over to Jobs. And we'll get started creating my job and then the tasks within that. So, I will click Jobs. And we'll choose to add a new job.

 

 

We'll go and just give this job a name, something super simple like myjob. And in the pool, we want to select the pool that we just created, which was mypool. So, with mypool selected here, I'll go in and click the Selected button. And now we can see I have myjob and mypool. That's all there is to creating the job because the real meat and potatoes of this comes in with the task. So, I'll click OK, go ahead and get the job created.

 

 

And once the job is created, we'll click on the job. And now we have the ability to add tasks to it.

 

 

You can see right now there are no tasks to display. We need to create those tasks. So I'll click Add a task, and we'll give these tasks an ID.

 

 

 

Now with tasks, you want to create multiple tasks so that it gets distributed across multiple nodes. You don't just create one task for multiple nodes. You create a task for basically each node. So, in this particular case, I'm going to create something like mytask. We're going to give it the display name mytask. And then the command line prompt that we're going to run since we're not running a specific application, we're just going to be running a basic command that's going to output some data to a text file, which is then going to be sent to the blob storage. The particular command that I want to run here is,

cmd /c "set AZ_BATCH & timeout /t 90> NUL"

 

This task literally launches a command prompt and just gathers generic information about the machine in the Azure batch processes running and then sits still for 90 seconds before it dumps the output out. This is just the first task. This is going to run on the first available node. It's not going to run on the second available node. So, keeping all of the rest of the information the same, I'm going to click Submit and then immediately recreate a second task with a different name but the same command so that we can see both tasks running concurrently on different nodes.

 

 

Now, under the Advanced Settings, if you scroll down, you can see that if we had additional files that needed to be set, environmental settings, task dependencies, meaning of other tasks had to be completed first or if other tasks depend on this, and application packages if we needed to run specific applications, we can configure these below. But I don't need to do any of that because this is just running a basic command line command prompt command. So, I'm going to click Submit. And then we'll create a new task again really quickly.

 

 

We'll call this mytask2. And this time, I'm just going to paste in the command to make this go nice and quickly. We'll scroll on down and click Submit again. Now we have two tasks cued up and ready to go. And what I should be able to do now is jump back into my batch account, and we should be able to see when the pool picks up this information and starts running these commands.

 

 

So back on to the batch overview pane, what I can do now is I can scroll on down to pools. And I should see, look, my allocation state is now steady. It's getting ready to start running my specific tasks once they become available. I'll click on mypool, and we can actually see a heat map of nodes that are currently going to be running.

 

 

Well, these two VMs, they're still in an idle state as they have completed the task. It went from starting the VMs to immediately running the tasks that had been queued up for it. And now that the task has completed after the 90-second pause that we built into the command, it's now sitting idle. What I can actually do is I should be able to go to these machines now and view the file that output the data.

 

 

Let's just go in and click on Nodes, and we'll pick the very first node that's right here on top of the list.

 

 

We'll go to Files. And it looks like one of these files right here on this node has 719 bytes.  Let's go ahead and give it a click and see what it says. I'll click the dot dot dot then view.

 

 

Look at that. This is the output of running that command prompt. So now we've actually seen how we can deploy pools, a job, and then tasks and then watch how it automatically spins up the volumes and starts running the tasks as they're queued up. That's how we can go about deploying an Azure Batch service from within the portal.

 

I hope this has been informative for you, and I'd like to thank you for viewing.