Introduction To Azure Durable Functions

Introduction

Durable Functions is an extension of Azure Functions. It enables us to write stateful functions in a serverless environment, and it allows us to define workflows in code. The extension lets us define stateful workflows in a new type of function called an orchestrator function. Here are some of the advantages of orchestrator functions:

• They define workflows in code. No JSON schemas or designers are needed.
• They can call other functions synchronously and asynchronously. The output from the called functions can be saved to local variables.
• They automatically checkpoint their progress whenever the function awaits. Local state is never lost if the process recycles or the VM reboots.

Let’s have some basic overview of Azure Functions and serverless. Then, we'll be in a position to understand how Durable Functions build on top of that, and why we might want to use them.

 
Function Chaining Pattern

Prerequisites

• C#
• Azure Functions

We should have knowledge of programming and it would be an advantage if we have experience in C# language. Next, the knowledge of Azure Functions is needed to create an Azure Function.

About Azure Functions

Azure Functions is Azure's Function-as-a-Service offering. It allows us to deploy individual functions, small pieces of code, that respond to various types of events. Each Azure Function has a trigger, that's what causes it to execute. And an example of triggers is timers. So, run a function every ten minutes, for example. Or another example of this is a message on a queue, run this code every time a message appears on a particular queue. And another very common example is an HTTP request. We can run a function every time a particular HTTP endpoint is called. And the functions themselves can be written in one of several different languages. Currently, there's support for C#, F#, JavaScript, and there's also more coming, such as Java and PowerShell.

In this article, we'll be focusing on C#, because now, Durable Functions do need to be written in a .NET language. But hopefully, that will change in the future, and we'll also be able to write our Durable Functions in JavaScript and other languages. Now, the Azure Functions Runtime also supports many different input and output bindings. And these are designed to make it easy to integrate with other services, like Blob Storage, Cosmos DB, and even things like SendGrid, for sending emails, or Twilio, for sending text messages. And of course, there's a whole lot more that could be said about Azure Functions.

 

About Serverless

Serverless refers to the idea that we want to be able to deploy our code, without having to worry about infrastructure. So, when we deploy our Azure Functions, we don't need to define how many servers that we want there to be, to execute those functions. The Azure Functions Runtime will automatically provision enough service, to handle whatever load we need. So, if we have a function that's triggered by a queue message, and that queue is currently empty, then they'll be no service at all, that we’ll pay for, to run our functions. But if that queue receives a message, then Azure Functions will ensure that there is an instance of our function running to process that message. And if our queue fills up with loads of messages, then Azure Functions might spin up many services, all running instances of our function, to work through that queue quickly. So, we get auto-scaling for free.

Another great thing about this is that it means we can use what's called a consumption-based pricing model, where we ‘are only paying for the time that our functions are actually running. And if our functions don't run at all, we pay nothing, unlike if our code was running on a virtual machine, which we'd have to pay for each month even if it was idle. Of course, there's a lot more that could be said about serverless architectures.

 

Why Durable Functions?

So now we've got a basic idea, of what Azure Functions is, and what we mean by serverless. So, let's move on, to see how Durable Functions fits into the picture. Well, it's very common for individual Azure Functions to be part of a larger workflow.

For example, we might have three functions that are called in sequence or chained together. Function One gets called first, and when it's finished, it triggers Function Two, and then when that's finished, it triggers Function Three. And, of course, it's possible to implement this with regular Azure Functions. We could use queue messages. So, Function One puts a message on a queue, and that message triggers Function Two, and so on. But what if we wanted to implement a fan-out, and fan-in pattern?

So, Function One might trigger multiple instances of Function Two, and then we want to wait until all of those instances of Function Two have finished, before triggering Function Three. But that's actually quite a lot harder to do, just with regular Azure Functions. Or, going back to the chaining example, what if we wanted to write an error handler, that could handle an error, wherever in the chain it occurred, whether it was in Function One, Two, or Three?

Again, that's quite tricky to implement, with regular Azure Functions. So, what Durable Functions allow us to do, is define our workflow in code. We write some very simple code in C#, that defines the workflow, what order to run the functions in. We can trigger parallel execution, and we can handle errors at any point in the workflow. And not only does this make workflows easier to implement, but they're also easier to understand because the relationship between all the functions is defined in a single place, in what's called the Orchestrator Function. Durable Functions also solves the problem of state for us. In a serverless environment, we need our functions to be stateless, but our workflow inherently has state associated with it. We need to track how far through the workflow we are so that we know what the next step in the process is. And typically, if we are doing this yourself, we'd end up storing that state in a database. But Durable Functions handles all state management transparently for us, so it becomes really easy to implement some quite advanced workflow patterns.

Benefits of Durable Functions

What are the key benefits of using Durable Functions, and why should we learn how to use them? Well, first of all, as we've explained, they let us define workflows in code. This means it's very easy to understand and picture of what the overall workflow is. It also provides a good separation of concerns, the code that defines the workflow, is independent of the code that implements each step in the workflow. Durable Functions simplify the implementation of complex workflows, such as fan-out and fan-in patterns, or waiting for human interaction. They also allow us to consolidate exception handling, into a single location, for the whole workflow. And they allow us to check on progress and request the cancellation of a workflow. And they also manage state view, keeping track of where we currently are in the workflow, without needing to create our own database tables. So, as we can see, there are lots of compelling reasons for us to use Durable Functions, for our workflows.

Concepts of Durable Function

We're going to be building lots of Durable Functions in this article series. But before we do so, we want to start off, by introducing a few of the most important key concepts. First of all, with Durable Functions, our workflows are defined within a new type of function, called an Orchestrator Function. And an Orchestrator Function has one role only, to define the workflow. It doesn't perform any actions itself, like calling APIs or writing to databases. Instead, it delegates all the actual steps in the workflow, to Activity Functions. When a new workflow is initiated, the Orchestrator Function is called, and it triggers the first activity. And then, the Orchestrator Function goes to sleep. And when the first activity finishes, the Orchestrator Function wakes up and carries on from where it left off, calling the next activity in the workflow. So, an Orchestrator Function calls an Activity Function, and an Activity Function is simply, a regular Azure Function, that participates in a workflow. Activity Functions can receive input data from the Orchestrator Function and can return data to it. And the way we start an orchestration is with an OrchestrationClient binding. Any regular Azure Function can use this binding, to start a new orchestration, or to get information about an orchestration. Let's look at an example, of how these various types of Azure Function work together, in a Durable Functions workflow.

Let's imagine, that we've got a regular queue-triggered Azure Function, and so it runs whenever a message appears on a particular queue. And then, in that function, it uses the OrchestrationClient binding, to trigger an Orchestrator Function. That Orchestrator Function might then call several Activity Functions, in order to implement the workflow. And these activities might be called in sequence, or they could be called in parallel.

Patterns of Durable Function

1. Function chaining
   Function chaining refers to the pattern of executing a sequence of functions in a particular order. Often the output of one function needs to be applied to the input of another function. Durable Functions allows us to implement this pattern concisely in code.
   
2. Fan-Out/ Fan-In
   Fan-out/fan-in refers to the pattern of executing multiple functions in parallel and then waiting for all to finish. Often some aggregation work is done on results returned from the functions. With normal functions, fanning out can be done by having the function send multiple messages to a queue. However, fanning back in is much more challenging. We'd have to write code to track when the queue-triggered functions end and store function outputs. The Durable Functions extension handles this pattern with relatively simple code.
   
3. Async HTTP APIs
   The third pattern is all about the problem of coordinating the state of long-running operations with external clients. A common way to implement this pattern is by having the long-running action triggered by an HTTP call, and then redirecting the client to a status endpoint that they can poll to learn when the operation completes.
   
   Durable Functions provides built-in APIs that simplify the code we write for interacting with long-running function executions.
   
4. Monitoring
   The monitor pattern refers to a flexible recurring process in a workflow - for example, polling until certain conditions are met. A regular timer-trigger can address a simple scenario, such as a periodic cleanup job, but its interval is static and managing instance lifetimes becomes complex. Durable Functions enables flexible recurrence intervals, task lifetime management, and the ability to create multiple monitor processes from a single orchestration.
   
5. Human Interaction
   Many processes involve some kind of human interaction. The tricky thing about involving humans in an automated process is that people are not always as highly available and responsive as cloud services. Automated processes must allow for this, and they often do so by using timeouts and compensation logic.
   

What is Event Sourcing?

Now we've said that an Orchestrator Function triggers Activity Functions, and then it goes to sleep, waiting for them to finish. But how does it store its state? When an Activity Function completes, how does the Orchestrator Function know, where it got up to, in the workflow? And the answer is, that behind the scenes, the Durable Functions extension is making use of an Azure Storage account. In that account, it uses storage queues, as a communication mechanism, to trigger the next functions. And it uses storage tables, to store the state of the orchestrations, or workflows, that are currently in progress. And the way that the state is stored, uses a technique called event sourcing. With event sourcing, rather than storing the current state, we store information about all the actions that led up to that state. And this approach has got some interesting benefits. One, is that it means we never need to update a row in our data store, we simply append new rows. And another one is that we have a full history of everything that happened, to get to this point, which is great for auditing, or diagnostic purposes.

So, the sequence of events, for a durable workflow, might look something like this. The Orchestrator starts, and schedules Activity One, and then the Orchestrator goes to sleep. And then Activity One runs, and when it completes, the Orchestrator wakes up, and Activity Two gets scheduled. And again, the Orchestrator then goes to sleep. When Activity Two completes, the Orchestrator wakes up again, and the Orchestrator, if that's the end of the workflow, can complete. So, whenever the Orchestrator Function wakes up because an activity has ended, it can replay through the stream of events, to work out where it had gone up to, in the workflow, and what should happen next.

What are Task Hubs?

When we configure Durable Functions, we provide them with a connection string, for a storage account, that it will use to store these events. And that storage account is under our control, so we can actually look inside the storage account, at the tables and the queues, using a tool like Azure Storage Explorer, and examine the contents. Now obviously, we're not meant to touch or manipulate this data in any way, but it can be a very useful way of understanding what's going on under the hood. And Durable Functions uses the terminology of task hubs, to refer to the storage associated with our Durable Functions. It's actually possible to have more than one task hub, and we can even put multiple task hubs into the same storage account if we want. However, for most use cases, it's more than sufficient, to just have a single task hub.

Ways to Develop Durable Functions

What do we need in order to be able to develop Durable Functions? Well, Azure Functions give us three ways to develop.

1. Azure portal: We can create functions directly in the Azure portal, and this is great for trying out Azure Functions, without installing anything on our development machine. So, it's great for experimenting, but we wouldn't recommend it for building production code.
2. Command line: We can develop from the Command line on any platform, just using our favorite text editor, like Visual Studio Code, and the cross-platform Azure Functions command line interface. And we've got the link here, for the tooling that we need to do that.
3. Visual Studio: We can use Visual Studio 2017. This comes with an Azure Functions extension, that will give us project and function templates, and we'll also benefit from the Great Intelligence, and debugging support that it offers.

And so, although we can use any one of these three techniques, to build Durable Functions, for the purposes of this articles, we are going to use Visual Studio 2017.

Set-up Durable Functions in VS2017

The good news is, that the free Visual Studio Community edition, can be used for creating Durable Functions, and we can download that from https://visualstudio.microsoft.com/vs/community/.

 

When we're installing Visual Studio 2017, do make sure that we select the Azure development workload.

 

Don't worry, if we've already installed Visual Studio without this, we can always go into the installer, select modify, and then add the Azure development workload later.

And this is going to give us several really nice Azure development extensions, as well as the Azure SDK. And two important things that we'll get, are the Azure Functions and WebJob Tools Visual Studio extension.

 

Here we can see it in the extensions list. And having this present will allow us to create new Azure Functions apps from the File-New Project menu.

We'll also get the Azure Storage Emulator. This is necessary if we want to test Durable Functions locally. And that's because, as we've explained, the Durable Functions extension makes use of Azure storage tables and queues, to implement its task hubs, for storing orchestration state and control messages. So, having the Storage Emulator enabled, will allow us to run and test our Durable Functions locally, against an emulated storage account.

Let's quickly see, how we can create a new Azure Function app, in Visual Studio.

Creating a First Project

In this demo, we'll use Visual Studio 2017 to create a new Azure Function app, and then, we'll enable Durable Functions, by adding the Durable Functions NuGet package to our project.

 

So here we are in Visual Studio 2017, and we've already installed the Azure development workload. If we go to File-New Project, we'll see that there's an Azure Functions template, that we can choose. We'll need to enter a name for our project and click OK, and that's going to create a basic function for us.

Now, it's going to prompt for the version of the Azure Functions Runtime, that we want to use. The original version of Azure Functions was based on the .NET framework. Now, we can also choose whether to create an empty function app or create one that's got a starter function. we're going to pick one with an HTTP function, just so that we've got something to test, to prove that it's working. And over on the right, we can see, that we can configure some settings for our function app. We can set up the storage account, that we'll be using, and by default, that's going to point to the Storage Emulator. Obviously, in production, we're going to be pointing at a real storage account. But for development purposes, this is fine.

 

So, we'll click OK, and it will take a minute or so, to create, but once it's done, we can see that we've got a fairly sparse project, that we've created. There's a local.settings.json file, and if we open this file, we can see the connection string, pointing at the Storage Emulator, that we can use, for local development and testing. And this vault doesn't get checked into source control, so when our function app is deployed to Azure, it will be getting these connection strings from our function apps app settings instead. And of course, there's our example function, Function1.cs, which is just going to respond with a hello message, whenever we call it.

How to Enable Durable Functions

We're going to add the Durable Functions extension, and we can do that by going into the Dependencies node and selecting NuGet.

 

Now, we're going to enable Include prerelease, and search for Durable Functions. And here we can see, that the Microsoft Azure WebJobs Extensions Durable Task NuGet package is available, and that's what we need to add. We'll be able to select 1.6.2, there will be newer versions that have been released.

 

So, once we've installed this NuGet extension to our project, we can see what it's done, by right-clicking on the project, and saying that we want to edit the csproj file.

 

And we can see, that all it's done is just added a new package reference, to the DurableTask extension.

Test the Function App

Let's just check that everything is working. We're going to run the created project.

 

and when we start debugging, it's going to launch the local Azure Functions host, and it will tell us the URI of our function.

 

In this case, it is a http://localhost:7071/api/Function1. So, let's copy that into the browser, and let's call the function.

As we can see, it has called our function, but we need it to pass a name parameter in the query string.

So, let's do that.

 

We're going to add name equals Kamlesh and going to call the function again. And now, we can see Hello Kamlesh. So, we're up and running.

See Also

One other important thing to note is that the Azure Durable Functions extension is open source. Read suggested related topics:

On GitHub: Azure Durable Function