Hot, Warm, and Cold data paths with IoT on Azure

by | May 25, 2021 | Technology | 0 comments

This article is contributed. See the original author and article here.

This year our winter season was one of the coldest I’ve ever experienced, bringing an array of winter related issues. One of the more costly winter issues I’ve seen are busted water faucets. I knew this was not something I wanted to face this year, so I started thinking about the preventable problem and a smarter solution than the traditional foam cover. After considering the design of a foam cover and its inability to fully protect against freezing temperatures, I constructed two insulated thermal units to cover each faucet of my home. Each unit contains an incandescent light to warm the surrounding surface wall of the water pipe connected to the faucet. To monitor effectiveness and alert of potential failures, I built an IoT based solution using an ESP8226 and Azure to monitor, store, and provide insight on the temperature from within the units.


 


To keep this solution lean and efficient, I went with a serverless architecture and implemented Hot, Warm, and Cold data paths. This serverless architecture makes use of serverless resources in Azure, Functions, IoT Hub, Data Explorer, App Service, and Signal R, to ease the management and cost of the overall solution. To learn more about the power of serverless services and architectures on Azure, look here.


 


If you’re not familiar with Hot, Warm, and Cold data paths, take a look at the following breakdown to understand the differences between each path as well as which Azure services in this solution enable them:


 


Hot Path



  • For processing or displaying data in real-time

  • Real time alerting and streaming operations are performed using this data

  • An Azure Function App, Signal R, and web app hosted on an Azure App Service are used here alert and stream data in real-time

    • Azure Function App provides a consumption based and elastic resource to ingest all incoming data for processing, alerting, and sending to Azure Signal R

    • Azure Signal R and the web app hosted on an Azure App Service enable the ability to stream data through a WebSocket based connection. 




Warm Path



  • For storing or displaying only a recent subset of data

  • Small analytic and batch processing operations are performed on this data

  • A web app hosted on an Azure App Service is used here as it can query and display the last 24 hours worth of temperature data per device from Azure Data Explorer 


Cold Path



  • For long-term storage of data

  • Time consuming analytics and batch processing is performed on this data

  • Azure Data Explorer is used here as it efficiently stores data for long periods of time, currently with a default of 100 years, and is an easy-to-use analytic engine, built on top of the Kusto Query Language (KQL)


 


Now each unit contains an ESP8266 with a DHT11 temperature sensor which can either send temperature data to my field gateway, IoT Edge running on a raspberry pi, or directly to Azure IoT Hub. This temperature data is then ingested, monitored, and displayed in real time using the following process:



niswitze_0-1621373297130.png


 


 



  1. An Azure Data Explorer instance ingests all temperature data for long-term storage (Cold data path)

  2. An Azure Function broadcasts all temperature data to an Azure SignalR instance (Hot data path)


    • This Azure Function also sends out a text alert if the temperature falls below a defined threshold


  3. The Azure SignalR instance broadcasts temperature data to all clients listening on a WebSocket based connection

  4. An App Service hosts a web app, displaying the latest temperature data record per device from the last 24 hours from Azure Data Explorer (Warm data Path)

  5. Finally, the web app creates a WebSocket connection to the Azure SignalR instance to receive temperature data in real time (Hot data path)


    • If set up correctly, the web app will look like the following:



niswitze_1-1621373297134.png


 


If you would like to recreate this solution, you can review my GitHub, link below, for instructions on how to set it up end-to-end.

https://github.com/niswitze/Hot-Warm-Cold-On-Azure-IoT


 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment