Process IoT Data with Azure Stream Analytics

Lab Overview:

This hands-on lab provides an Azure administrator with an understanding of how to configure an IoT Hub to consume IoT device telemetry, create an Azure Stream Analytics job to manage the data, and display the output in a useful format for monitoring and reporting. In recent years, there has been a surge in the number of IoT devices connected to corporate networks. Obtaining the skills to manage and monitor these devices actively can be useful in many career paths, especially for Azure Administrators and NOC/SOC Analysts.

Understand the scenario

You are a system administrator for a company that is rolling out an Internet of Things (IoT) data solution. You need to set up Azure Stream Analytics to process and route IoT device telemetry. You will create an Azure Stream Analytics job and define an Azure IoT Hub and an input. You will configure an Azure Storage Table as a Stream Analytics output. You will then define a query to load the table with the average readings coming in from the Azure IoT Hub. Finally, you will verify your results through a test web site. To complete these tasks, you will use an Azure resource group that contains a preconfigured web app, an IoT Hub, and a storage account.

Configure IoT Hub:

The Azure lab environment is provisioned with an IoT Hub, a Web App, and a Storage Account. For the first part of this lab, you will capture data used in later sections (i.e., an IoT hub permission connection string primary key and a string for configuring an endpoint for events). Then you will provision the IoT devices that will be monitored in your environment. You will simulate collecting events from a building, a vehicle, and a piece of equipment.

Provision Azure Stream Analytics:

Microsoft Azure provides Stream Analytics, a real-time event processing engine for managing, monitoring, and performing use-case driven analytics on streaming data from your IoT (and other types of) devices. For this task, you will create a Stream Analytics job, configure a stream input, and add the type of output you desire (in this case, datastore table storage).

Generate Sample Data:

Microsoft Azure provides an IoT Test Page so that you can test your Stream Analytics job by emulating device interactions. The test page is a website that allows you to apply multiple tests, such as to emulate devices, verify device data capture, send messages to devices, verify stream analytics, etc. The device emulation within the web app is based on the same IoT code that would be used on an actual IoT device, and the emulation interacts with the Azure IoT hub in the same way as an IoT device would.

Start the Stream Analytics Job:

After you’ve tested the ingestion of data, you will now configure your Stream Analytics Job to work with the IoT logs. For this task, you will configure a test query to make sure that the data is accessible, then you will refine the query to output the data in a useful format for monitoring. This lesson will provide exposure to the query language used within Stream Analytics, which is SQL-like.

Test the Stream Analytics Job:

For this task, you will generate more test data, using the IoT Test Page. You will first emulate devices and verify the IoT telemetry data is generated. Then you will select “verify stream analytics” and observe the output. The output will display relevant aggregate data based on the analytics query you created.

Lab Summary Conclusion:

In this hands-on virtual lab, you will learn how to configure an Azure IoT Hub and Stream Analytics to manage the input and output of IoT device telemetry data. These skills will allow you to understand how Azure can be used to manage, monitor effectively, and analyze your IoT environment. This capability can enable you to perform analytics for issues that may arise in your environment, such as device maintenance requirements, security issues, misconfigurations, or other critical insights. These skills are essential for someone pursuing a career in the IoT/OT space and correlate to the skills needed for IT security operations. These skills are valuable to an Azure Administrator and an IT/IoT/OT NOC or SOC Analyst.

Other Challenges in this series

• Guided Challenge - Define IoT Hub Endpoints and Consumer Groups
• Advanced Challenge - Can you Process Incoming IoT Data in Real-Time?