Storing and Visualizing IoT Data with Azure Time Series Insights

Introduction

 

In this article, we are going through how to create a new Time Series Insights environment of our own and link it to our hub. We'll even go through a hands-on example of using Time Series Insights to explore IoT data as we work to troubleshoot a fictional hardware failure.

 

Github Code Link

 

https://github.com/Usman-Sheikh/Azure-Time-Series-Insights-Code

 

Road Map

1. Azure Time Series Insights Overview
2. Creating a Time Series Insights Environment
3. IoT Hub with TSI in Action

Azure Time Series Insights Overview

 

Azure Time Series Insights, which you'll often see abbreviated as just TSI, is an all-in-one solution that combines storage, analytics, and visualization. Let's talk about storage first. Time Series Insights is a hosted Platform as a Service solution. There is no storage infrastructure for you to maintain. Microsoft does that for you. You can scale your Time Series Insights environments up and down to meet your needs. How much data can you store with Time Series Insights? You can push billions of events to Time Series Insights, and more importantly, still retrieve them quickly. How long will Time Series Insights keep your data for? Well, that's up to you. You can configure a retention policy that fits the needs of your IoT solution. If you only need recent data, you can configure it to discard data after a few weeks, and if you need to hang onto your data for longer, Time Series Insights is now capable of storing up to 400 days’ worth of data. And if that is still not enough, the preview version of Time Series Insights also includes cold data storage, which gives you basically unlimited data retention. Another key feature of Time Series Insights is ad-hoc analytics. You can use Time Series Insights Explorer to analyze data from your IoT devices. Time Series Explorer is a full-featured hosted tool for analyzing your data, but if you want to explore things at a lower level, Time Series Insights also features a powerful API for querying your data. Time Series Insights queries are written using JSON.

 

Time Series Insights Explorer allows you to visualize your data in numerous ways right out-of-the-box. You can view your data across datasets, apply filters, and drill into the raw data. And if you don't want to use Time Series Insights Explorer, you can actually embed the Time Series Insights visualizations into your own applications, or execute queries directly using the REST API and then visualize those results using the tools of your choice.

 

Most developers are already quite comfortable with relational databases, and you might be tempted to use SQL Server to store your IoT data. But while SQL might play a role in your solution, you probably don't want to store all of your raw data in there, especially if your IoT solution is going to be dealing with millions or billions of data points over many years. SQL Server is designed for storing relational data, not time series data. I can tell you from personal experience if you try to store even moderate amounts of your raw time series data in SQL, you're going to have a bad time. SQL, again, maybe a component of your IoT solution, but you'll want to pick a more suitable data store such as Azure Time Series Insights for storing and querying the raw data.

 

Azure Time Series Insights and IoT Hub

 

Time Series Insights pulls data from IoT hub using a mechanism that involves the existence of a consumer group. You can think of a consumer group as a view of the data that a hub has received.

 

For more on consumer groups check here.

 

Creating a Time Series Insights Environment

 

There are a couple of ways we could create a Time Series Insights environment. I'm going to use the Azure portal, but we could do the same thing using the Azure CLI as well.

 

We’re going to start by creating a Resource group. If you wish to know how to do that, you can find it here. Once you have created a resource group add an IoT hub. Since that is not a part of this article, you can follow the method for it here. After that, we will add a time series insights resource.

 

 

 

 

 

Give your environment a name that is unique to it, the resource group that was previously made and all the other required information. Then click on the Event Source button next to Create. Now you will have to configure the TSI resource. Do it as follows, filling in with information relevant to you. Make sure to add a new consumer group for your environment. Click on the Add button once you’ve typed a name and wait for a few seconds so that the consumer group is created.

 

 

 

Click on Review + Create.

 

 

 

Finally, create a TSI resource. Wait for it to deploy.

 

Once the TSI resource is deployed go to it. At first, you will see this on the resource page.

 

 

 

What this means is that we won't actually be able to view anything in our environment yet, because we haven't defined any access policies. We can do that by clicking Data Access Policies here under Environment Topology. Now let's click Add, and let's define a new policy.

 

 

 

First, we need to select the user. I'm going to give myself access to the environment. Now we need to set a role. I'm going to give myself full permission to the environment.

 

 

 

 

Let's click OK.

 

So if we go back over to the overview, yep, looks good. The banner is gone. That means our environment is ready for use.

 

Now we’re going to run our virtual device so that we can push some data up to our hub, and over here to Time Series Insights.

 

IoT Hub with TSI in Action

 

Let's see what our new Time Series Insights environment can do. In this demo, we're going to use a new sample application that I'm providing to simulate a machine sending telemetry data up to our hub. If you want to follow along, you'll need to download the exercise files for this course or clone the GitHub repository at this URL. Inside, you'll find the machine-sample project that we'll be executing. This virtual machine will push up values for four different sensors, temperature, oil pressure, coolant level, and vibration level. Remember these sensors, because these will be important later, as you'll soon see. Our machine-sample is configured to connect to our hub using a device connection string. So first things first, let's add a new device, and get the connection string for it. Go to your IoT hub service > IoT Devices (appears in the sidebar) > Add and you will arrive at this screen. Name your device whatever you wish to. I named mine machine-sample-01. Then click on save.

 

 

 

Once your device is created it will appear on the IoT hub > IoT Devices page. Click on the hub you just created and you will see its details.

 

 

 

Copy the connection string. Now that we have our connection string, we're ready to run our sample application. Let's go over to our terminal. I'm already in my machine-sample project folder. If you aren't there yet, go ahead and change to that directory. Running this sample is actually pretty easy. We just execute the dotnet run command and pass in our device connection string as the argument, but again, just like last time, be sure you wrap that connection string in quotation marks, like so. And there we go, our machine is off and running. Let's allow our machine to run for 5 minutes or so. Then we will have enough data to make this interesting.

 

 

 

What happens after you execute this program is this,

 

 

 

Although this is not a cause for concern that much, the purpose of this explosion is to explain how we can use Time Series Insights in order to explore why such a catastrophic event occurred. Let’s go to our TSI service and explore the problems. Go into resource groups, and then into our current resource group, and then finally, let's go into our Time Series environment. Be sure you click the environment, not the event source. Here from the overview, we can click Go to Environment to launch Time Series Insights Explorer.

 

 

 

This takes us to the Time Series Insights explorer.

 

 

 

There's a lot we can do with Time Series Insights Explorer. I'm going to give you a quick crash-course overview. Our workspace is divided into three sections. On the left, we have our query options.

 

 

 

On the right, we have a visualization of our current query:

 

 

 

And down below, we can see the raw events or statistics from our result set.

 

 

 

Our device didn't run very long, so let's drill in by dragging the selection around our events. We can then right-click the selection and choose zoom.

 

 

 

 

The number of events isn't really helpful here, so let's change our query. On the left, instead of Events, let's have it graph the Values.

 

 

 

Okay, this looks different, but it's still not all that helpful. That's because it's graphing all the sensor values together, and not distinguishing between our different sensors. We can fix that by having it split the value by SensorName.

 

 

 

Now we're getting somewhere.

 

 

 

Let's drill in a little bit further so we can see some more detail.

 

 

 

 

It looks like everything was operating normally at first. Our Temperature, our OilPressure, our CoolantLevel, and our Vibration sensor were all reporting steady values. But if we focus just on our vibration sensor, we can see that something interesting happened right about here. There was a big spike in vibrations just for a moment.

 

 

 

So there's our big spike in vibrations, and after that our coolant level begins to decline. And if we expand our selection just a little bit here, we can see that as our coolant level declines, our temperature and our oil pressure both also start to climb, as well as our vibrations. We can see that these values continue to increase more and more, until finally, everything just stops, and that's probably the point where our machine exploded. So what happened? It looks like that something on our machine failed right here around this time.

 

 

 

Whatever failed, that somehow seems to have caused a leak in our coolant here. As the coolant level dropped, our temperature, our oil pressure, and eventually our vibrations, all began to increase, and that eventually led to our catastrophic failure. This is a fictional example, of course, but it does illustrate how we can use Time Series Insights and TSI Explorer to perform a root cause analysis. This process we went through would be mostly the same with a real IoT solution and that's because Time Series Insights is built for this type of analysis on an IoT scale with time-series datasets. As a quick reminder before we wrap up, a Time Series environment isn't cheap, so don't forget to remove your environment once you are finished playing around with it.

 

Summary

 

In this module, we learned about Azure Time Series Insights. We looked at its three major features, storage, analytics, and visualization. We also talked about why we need a solution like Time Series Insights, and why a more traditional solution, like maybe a relational database, might not be a good fit for some IoT solutions. We then worked together to provision a new TSI environment and we connected it to our IoT Hub. Finally, we worked through a fictional example of using Time Series Insights Explorer to determine what caused a piece of virtual equipment to fail.