Handling Live Data Feed Using Azure Stream Analytics Job

Azure Event hubs are the best method for ingesting events in steam analytics. The demo application generates events and sends them to an event hub. If you are new to event hubs, please refer to my previous article about it.

Next, you will have to create an Event Hub namespace and event hub. Since there are tons of resources available on the web on how to perform this, I am going to skip you showing steps to create a namespace and event hub. Make sure you note down the connection string from shared access policies -highlighted in the below snapshot.

Initiate the Twitter client application

For the application to receive the tweet events directly, it needs permission to call on the Twitter streaming APIs. We will walk through that in the following steps,

Configuring our application

For the sake of simplicity, I will not be explaining how to create a Twitter app here, one thing to note is you have to copy the consumer API secret key and consumer API key once when you create the app.

Before starting our application which collects tweets using Twitter API about a topic, we have to provide certain info including conn keys and Event hub conn strings.

• Download TwitterClientcore application from GitHub save in a local folder
• Edit the app.config file in the notepad and change the appsettings to suit your connection string values
• After completing the above changes go to the directory where the Twitterclientcore app is saved. Now build the project using ‘dotnet build’ command post which starts the app using the command ‘dotnet run’ and then you can now see the app starts sending tweets to your event hub.

Download dotnet core SDK from the Microsoft site and install it in your system. I downloaded the .NET core 3.1 SDK installer from here.

dotnet build

dotnet run

Now that we could see the app is sending tweets into the eventhub successfully, we can proceed with setting upstream analytics.

Stream Analytics

Create a new ‘Stream analytics Job’ resource in your respective resource group.

Create Input

Once created when you go to your resource you could see it is currently in a ‘Stopped’ state which is normal. The job can be started only after proper inputs/outputs have been configured which we are going to perform now.

In our newly created stream analytics job, from the left menu select ‘Inputs’ under the category ‘Job Topology’.

If you face any connection test failure when creating this make sure you have selected ‘Connecting string’ as auth mode with the following options.

It's now time to create the input query. Go to overview and click the edit button on the query box below,

For testing I am providing it with my simple query “select * from streaminput2” -remember streaminput2 is the name we have given while creating the inputs of the stream analytics job. Once you have typed the query you can see the output in the ‘input preview’ pane below.

You can also click the test query button with a different query and see the test results in the next pane below.

Create Output

So far we have created an event stream, event input, and query to transform the incoming data over the event streams. One final step is to configure the outputs for the stream analytics job.

We can push the results to Azure SQL database, Azure BLOB Storage, PowerBI dashboard, Azure table storage, and even to Event Hubs, etc., based on one's need. Here I will demo pushing to Azure BLOB.

Now as we have successfully configured Input, Query, and Output, we can go ahead and start the Stream Analytics Job. Before starting you have to make sure the Twitter client application which we started earlier is running.

We could see the data is getting stored as JSON files in blob storage, we could see the size of the files getting increased gradually once if we refresh which proves the data that is getting stored.

Summary

Thanks for staying this long for reading this article. I would always like to have it short but given this is an End-to-End project it took a while -kudos to you all!

What next?

We have introduced ourselves to Azure Stream Analytics through which we can capture live stream data to blob storage easily. In the future, we will see how to save live feed data into an SQL database and then create a real-time PowerBI dashboard from a live Twitter feed which I believe will be interesting topics.

References

Microsoft official azure stream analytics documentation