Azure batch task monitoring and cost saving

In this blog we will see Montior tasks on Azure batch and remove vms for cost saving. Check task state to print stdout of task.

Azure Batch has come out of preview and its technical overview gives details of how it works. More interesting and detail article is provided here and also gives a brief description of Pool and VM Lifetime. The cost charged will be based on number of minutes number of VMs are running in a pool. It talks about two ways to maintain a pool. One is pre-defined and other is dynamic with dynamic aimed at cost efficiency. There are couple of examples posted also on MSDN pages that detail with Pool Creation and then tasks. Other examples that talk about Autopool creation through workitem.

A similarity in these code is using toolbox and wait for predefined period of time for tasks to complete. Below is the code snipped on it where a task monitor is created and wait for 10min.

client.OpenToolbox().CreateTaskStateMonitor().WaitAll(job.ListTasks(), TaskState.Completed, TimeSpan.FromMinutes(10));

Then stdout and stderr of the tasks are read and displayed in console.

foreach (ICloudTask task1 in job.ListTasks())

{

   Console.WriteLine("Task " + task1.Name + task1.GetTaskFile(Constants.StandardOutFileName).ReadAsString() + " :\n");

}

We are going to extend this code for reading the state every 10min, take care of not reading the stdout when task is not complete, print only once per task and remove the vm of the task that is complete.

First we need to add check to above code to avoid exceptions due to our attempt to read a task output before its done. Add a condition inside the for loop to avoid reading the task output before its complete

Next we also add a while loop to handle keep reading the tasks status till all tasks are done. In the same code will remove the vm of task which is complete to save cost

Next to keep the console clean, we do not want to keep reading every task output, so we will use a dictionary to maintain list of tasks that where completed previously and not read them again.

//Get the job to monitor status.

ICloudJob job = wm.GetJob(WorkItemName, JobName);

Console.WriteLine("Wait 10 minute for all tasks to reach the completed state");

//Get PoolName of workitem

string workItemPoolName = job.PoolName;

IPoolManager pm = client.OpenPoolManager();

ICloudPool pool = pm.GetPool(workItemPoolName);

Dictionary < string, TaskState > taskState = new Dictionary < string, TaskState > ();

while (taskState.Count < tasksToAdd.Count)

{

    client.OpenToolbox().CreateTaskStateMonitor().WaitAll(job.ListTasks(), TaskState.Completed, TimeSpan.FromMinutes(10));

    foreach(ICloudTask task1 in wm.ListTasks(WorkItemName, JobName))

    {

        //Check in dictionary if task as already completed in previous read

        if (!taskState.ContainsKey(task1.Name))

        {

            if (task1.State == TaskState.Completed)

            {

                Console.WriteLine("StdOut for Task " + task1.Name + task1.GetTaskFile(Constants.StandardOutFileName).ReadAsString() + " :\n");

                Console.WriteLine("StdErr for Task " + task1.Name + task1.GetTaskFile(Constants.StandardErrorFileName).ReadAsString() + " :\n");

                //add to dictionary

                taskState.Add(task1.Name, task1.State);

                //Get the VM of task and remove. To be sure, set deallocation as taskcomplete

                VMInformation vmoftask = task1.VMInformation;

                Console.WriteLine("Deleting vm: " + vmoftask.VMName + "\n");

                IVM vmToDelete = pool.GetVM(vmoftask.VMName);

                vmToDelete.RemoveFromPool(TVMDeallocationOption.TaskCompletion);

                Console.WriteLine("Deleted vm: " + vmoftask.VMName + "\n");


            }

            //Jsut print the task state if its not compplete

            else Console.WriteLine("Task " + task1.Name + " is at" + task1.State + " state \n");

        }

    }

    Console.WriteLine("Wait again 10 minute to get all tasks states. \n");

    //Get latest Job object again

    job.Refresh();

}