Post
Ask Question

Azure Function As Output Job Topology Of An Azure Stream Analytics Job

Introduction

 
For the last few days, I have been playing with my MXChip Azure IoT Dev Kit. In this article, we are going to see how we can set up an Azure Function as an Output job topology of an Azure Stream Analytics job. Doesn’t that sound interesting? In our previous articles, we have already seen what an Azure Stream Analytics Job is and how can we create it by using the portal and Visual Studio. If you haven’t read those articles, I strongly recommend you to read them. Let’s jump into this article now.
 
You can always read this article on my blog here. 
 
Background
 
As I mentioned earlier, in this article we will be,
  1. Using our existing Azure Stream Analytics job.
  2. Creating a new Azure Function App.
  3. Setting up the newly created Azure function as an output job topology of the stream analytics job.
  4. Monitoring the data coming to the Azure Function from the stream analytics job.
Play with Azure Functions
 
Yeah, we are going to play with it. Let’s go and create one then.
 
Creating an Azure Function
 
To create an Azure Function application, you need to log into your Azure portal and click on the "Create a resource" icon, and then, you can search for the ” Function App”.
In the next screen, provide the following information.
  1. App Name
  2. Subscription
  3. Resource Group
  4. OS
  5. Hosting plan
  6. Location
  7. Runtime stack
  8. Storage
  9. Application Insights
Here, the Consumption Plan Hosting plan allows you to pay per execution, and the App Service plan allows you to have a predefined capacity. For the runtime stack, we will use .NET, however, you are free to use anything you wish.
 
Once you have created the same, you should be able to see it under the Function Apps section.
 
Creating an Azure Function Solution and Function
 
Now, let’s go to our Visual Studio and create a new solution for our Azure Function.
 
Azure Function Project Type 
 
Azure Function Project Type
 
Now, you can right-click on your newly created project and add a new HttpTrigger Function. We will keep the access rights to anonymous for now. I have named my function as “GetData”. For now, let’s just get the data from our Stream Analytics job and just check the length.
  1. using Microsoft.Azure.WebJobs;  
  2. using Microsoft.Azure.WebJobs.Extensions.Http;  
  3. using Microsoft.Extensions.Logging;  
  4. using Newtonsoft.Json;  
  5. using System.Net;  
  6. using System.Net.Http;  
  7. using System.Threading.Tasks;  
  8. namespace ml.IoTPlatform.AzureFunctions {  
  9.     public static class GetData {  
  10.         [FunctionName("GetData")] public static async Task < HttpResponseMessage > Run([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestMessage req, ILogger log) {  
  11.             log.LogInformation($ "GetData function triggered with Uri {req.RequestUri}");  
  12.             string content = await req.Content.ReadAsStringAsync();  
  13.             log.LogInformation($ "String content is {content}");  
  14.             dynamic data = JsonConvert.DeserializeObject(content);  
  15.             log.LogInformation($ "Data count is {data?.Count}");  
  16.             if (data ? .ToString() ? .Length > 262144) {  
  17.                 return new HttpResponseMessage(HttpStatusCode.RequestEntityTooLarge);  
  18.             }  
  19.             return req.CreateResponse(HttpStatusCode.OK, "Success");  
  20.         }  
  21.     }  
  22. }  
As you can see, we are not doing anything much for now; we are just receiving the data as HttpRequestMessage and reading the content as req.Content.ReadAsStringAsync() and then deserializing the object. If you are not doing this step, you may get an error as “No MediaTypeFormatter is available to read an object of type ‘Object’ from content with media type ‘application/octet-stream’.“
 
We are also checking the entity length and if it is too large, we are sending a HttpResponseMessage with status code 413.
 
Publish the Azure Function App
 
To publish your Azure Function app, just right-click on your project and click "Publish". Then, set up your publishing target by choosing the existing Azure Function App. Remember, we have created one earlier? Once you publish the same, you can go into your Function app and see your Function. You can also test the same with some dummy data.
 
There are probabilities to get an error as “Web Deploy cannot modify the file on the destination because it is locked by an external process” when you try to publish your Function App from Visual Studio, while your Function App is running. To fix this, you can see my answer here.
 
Function App in Portal 
 
Function App in Portal
 
Azure Stream Analytics Job
 
Let’s go back to our Azure Stream Analytics as we have already configured our Azure Function App successfully.
 
Configure Azure Function Output
 
In my previous article, we created an Azure Stream Analytics job solution using Visual Studio. Let’s open that solution now and configure the new output for Azure Function.
 
Solution Explorer 
 
Solution Explorer
 
While configuring the Azure Function output, please make sure that you are selecting the existing Azure Function app.
 
Azure Function Output Configuration 
 
Azure Function Output Configuration
 
Update the Script
 
We should also do some changes in our Script.asaql file to support our newly created output.
  1. WITH basicoutput AS   
  2. (   
  3.          SELECT   messageid,   
  4.                   deviceid,   
  5.                   temperature,   
  6.                   humidity,   
  7.                   pressure,   
  8.                   pointinfo,   
  9.                   iothub,   
  10.                   Max(eventenqueuedutctime) AS eventenqueuedutctime,   
  11.                   eventprocessedutctime,   
  12.                   partitionid   
  13.          FROM     input TIMESTAMP by eventenqueuedutctime   
  14.          GROUP BY tumblingwindow(second, 10),   
  15.                   messageid,   
  16.                   deviceid,   
  17.                   temperature,   
  18.                   humidity,   
  19.                   pressure,   
  20.                   pointinfo,   
  21.                   iothub,   
  22.                   eventenqueuedutctime,   
  23.                   eventprocessedutctime,   
  24.                   partitionid )   
  25. SELECT *   
  26. INTO   sqlserveroutput   
  27. FROM   basicoutputSELECT *   
  28. INTO   azurefunctionoutput   
  29. FROM   basicoutput  
Updating the TLS Version
 
Once that is done, just click "Submit to Azure". If you have any doubts in this section, read my previous posts on this topic. Now, let’s log into the portal again and check if all the outputs, inputs, and the query are published or not.
 
Outputs in Portal 
 
Outputs in Portal
 
Cool! Well done. It seems like it is published. Now, if you click on the AzureFunctionOutput, you may get a warning as “Please make sure that the Minimum TLS version is set to 1.0 on your Azure Functions before you start your ASA job“.
 
I would rather treat this as an error instead of a warning because without making these changes, our Azure Stream Analytics job will not write to our Azure Function. So, this is very important. I spent many hours on this and finally found that this warning was the root cause of my issue. You can see my explanation about this here.
 
So, just go to your Azure Function app and click on Platform Features -> SSL -> Minimum TLS Version.
 
Setting TLS Version 
 
Setting TLS Version
 
There is a saying that "developers don’t care about warning but only the errors". Well, in some cases, it is true. Hmm... I was just kidding.
 
Output
 
Once you have done everything mentioned, you are good to go and start your Stream Analytics job. Please make sure that your MXChip is connected to a power source so that the device can start sending the data.
 
Checking the SQL Server Output
 
Now, let’s log into our SQL Server database and run the below query to make sure that we are getting the data from the device.
  1. SELECT TOP (1000) [id],   
  2.                   [messageid],   
  3.                   [deviceid],   
  4.                   [temperature],   
  5.                   [humidity],   
  6.                   [pressure],   
  7.                   [pointinfo],   
  8.                   [iothub],   
  9.                   [eventenqueuedutctime],   
  10.                   [eventprocessedutctime],   
  11.                   [partitionid]   
  12. FROM   [dbo].[streamdata]   
  13. ORDER  BY [eventenqueuedutctime] DESC   
SQL Server Output Data 
 
SQL Server Output Data
 
Checking Azure Function Output
 
To check the Azure Function Output, we can go back to our Azure Function, click on the Function, and use the Monitor option.
 
Azure Function Output Data 
 
Azure Function Output Data
 
Please note that you can always check your Azure Stream Analytics job Activity Log if you have found something not working.
 

Conclusion

 
In this article, we have learned how to,
  1. Work with an Azure Stream Analytics job
  2. Create an Azure Function App
  3. Create an Azure Function App solution in Visual Studio
  4. Write an HttpTrigger function and publish the same to the Azure Function App
  5. Set up the Azure Function App as an output job topology of Azure Stream Analytics job
  6. Use the created package in another solution
In our next article, we will see how we can send this Azure Function Output data to an Azure SignalR service and then get the same data in an Angular application. I can’t wait to write my next article.
 
Your turn. What do you think?
 
Thanks a lot for reading. I will come back with another post on the same topic very soon. Did I miss anything that you may think is needed? Did you find this post useful? Kindly do not forget to share your feedback.


Similar Articles