Building a Real World Health Condition (Comorbidity) Dashboard with Azure Databricks and Delta Lake

by Contributed | Dec 16, 2020 | Technology

This article is contributed. See the original author and article here.

This post was authored by Bruce Nelson, Senior Solutions Architect at Databricks and Clinton Ford, Staff Partner Marketing Manager at Databricks

Overview

Healthcare organizations are improving the patient experience and delivering better health outcomes with analytic dashboards and machine learning models on top of existing electronic health records (EHR), digital medical images and streaming data from medical devices and wearables. Azure Databricks and Delta Lake make it easier to work with large clinical datasets to identify top patient conditions.

Using Delta Lake to build a comorbidity dashboard

Simulated EHR data are based on roughly 10,000 patients in Massachusetts and generated using the Synthea simulator. Our ETL notebook ingests and de-identifies our data, then prepares it for our visualization notebook. We create visualizations and a simple dashboard that show the top conditions (comorbidities) in our real world data and also analyze the correlation between any two conditions specified by the user.

Extract, transform and load (ETL)

To begin, we use pyspark to read EHR data from comma-separated values (CSV) files, de-identify patient personally identifiable information (PII) and write to Delta Lake for analysis. Using Delta Lake is a best practice for ingestion, ETL and stream processing as an open source format with support for ACID transactions, faster processing with Delta Engine and easy integration with other Azure services for additional use cases.

EHR data analysis and comorbidity dashboard

In this notebook we visualize top conditions in the database and create a simple dashboard to analyze the correlation between any two conditions specified by the user. You can share this notebook as a dashboard following these instructions.

Next steps

For additional background on this use case see this blog post. See live demos or get hands on at an Azure Databricks event. Go even deeper with this 3-part webinar training series to operationalize machine learning models for your own organization.

CI CD in Azure Synapse Analytics Part 2

by Contributed | Dec 16, 2020 | Technology

This article is contributed. See the original author and article here.

Hello Dear Reader, when last we were together in CI CD in Azure Synapse Analytics Part 1 we started out our series by:

• Creating an Azure DevOps project


• Linking our Azure Synapse Analytics environment to that Project via Git


• Validating that our Azure DevOps Repo was populated with our Azure Synapse Analytics environment

This time we will:

• Create a new branch on our Repo


• Edit our Azure Synapse Analytics environment
  
  
  
  • Specifically my SQL scripts have demos all over the place and Buck Woody said I have to clean up my very messy room …. Azure Synapse Analytics environment
  
  
  
  


• Create a Pull Request in Azure Synapse Analytics to merge our new branch with the main


• Approve the Pull Request in Azure DevOps


• Validate our main branch is updated in our Azure Synapse Analytics Environment

We will start by opening up our Azure Synapse Analytics environment by navigating to https://web.azuresynapse.net and signing in, and then navigating to the Develop blade.  We will then click on the down arrow next to the main branch and select + New Branch.

Our DevOps project stores SQL scripts, Pipelines, Notebooks, Linked Services, and many other items from our Azure Synapse Analytics environment.  It also stores meta-data. Today we will make pure meta-data changes by creating folders to organize our SQL Scripts.  So we will create a working branch that describes our change and name it Folders.

Looking at the Develop blade we can see that the main working branch has been supplanted by the Folders branch.  We will click on the three ellipsis … next to SQL scripts and create a new folder. 

Now we create our folders.  We will repeat this step for each folder that we will create.  The first folder will be for our SQL Serverless demos.  Now a quick song montage while I create the rest of the folders for us.

Before we start organizing we will take a before picture.  Here is a good look at all of my SQL Scripts without any folders.

These demos are all over the place!!! We start by hovering over one of the SQL Scripts and then clicking the three ellipsis … by the script and selecting Move to. 

We will select the College demos folder for the 01 Create College Tables script, and then click move here.  One more quick song montage while I organize all of the scripts for us.

Now that all of the scripts are organized there are some very IMPORTAINT things to point out.  Each script has a dot next to the name.  This signifies that the changes to our Folders branch have not yet been committed.  We need to click on Commit all in order to save these.  

If we navigated off of this page prior to committing the change all of the folders we created and the organization we’ve done would be gone.  

If we were writing a new SQL script, Notebook, or Pipeline each of those items have a Commit button in the user interface.  That will commit just that item as you work.  You can also use the Commit all button if you want to commit all changes you’ve made to the branch.  The good news is we have options!

So now we commit and preserve these changes.

After committing notice all of the dots have gone away.  This means our commit to branch Folders was successful.

Now we go to the Folders Branch click the drop down button and click Create pull request.  

This will launch Azure DevOps and bring us to a Pull Request window.   Now is a good moment for a quick sidebar.

A few items <begin sidebar>:

1. I’m an army of one right now


2. In your environment you should have a sprint you are working on, and a work item associated with the work you are doing.  That would get entered into the pull request for me (see item 1)


3. You should have Reviewers identified for your code, and you would select them here, for me (see item 1)


4. When you create this other processes would be in place, for the sake of the tutorial I will show you everything because….. (see item 1)


5. </end sidebar>

We create a title stating what we are doing “Folders Merge”, in the description we give a little more info.  We do not have Reviewers or a Work Item, remember the sidebar, then we click Create.

We will approve and then Complete these changes.  This will merge the Folders branch with the main branch and the Folders branch will cease to exist.

Navigating back to our Azure Synapse Analytics environment we can see we are in the main branch and all of our beautiful folders are in place.

Next in our series we will migrate all of these changes to a new squeaky clean QA environment by creating an Azure DevOps pipeline to create an artifact and then a release pipeline to deploy our artifact to a new Azure Synapse Analytics QA.

Thank you for stopping by!

Combining Azure Sphere IoT security with Azure RTOS real-time capabilities

by Contributed | Dec 16, 2020 | Technology

This article is contributed. See the original author and article here.

Learn how you can combine the best of Azure Sphere IoT security with the power of Azure RTOS to deliver new IoT solution categories. Covered in this article is a brief introduction to Azure RTOS, Azure Sphere, and the MediaTek MT3620 MCU architecture. You will also learn why you might want to run real-time applications on an Azure Sphere, how applications communicate across cores and how to get started.

Azure RTOS

Azure RTOS is a Real-Time Operating System available from Microsoft that runs on most microcontrollers. Azure RTOS  includes a small powerful Real Time operating system called ThreadX, a GUI designer plus GUI library, and more.  Azure RTOS ThreadX makes it easier to build reliable embedded solutions with a rich set of services including real-time, multithreading, inter-thread communication, synchronization, timers, and memory management.

Azure RTOS is free to use on an Azure Sphere MT3620 MCU and is covered by the “Distribution and Production Use” license.

Azure Sphere

The traditional approach to IoT microcontroller development is that you are responsible for integrating and maintaining all the bits of a solution. This includes the communications stack, security, authentication, certificates, identity, encryption libraries, update processes, your solution and more. Not only are you a solution domain expert, but you are now responsible for tracking new and emerging security threats, mitigating, and updating to protect IoT devices.

Azure Sphere is a unique highly secure IoT platform. You focus on your solution, Azure Sphere deals with security, identity, certificates, reporting, tracking emerging attack vectors, mitigating, updating the platform, and application distribution to protect your solutions, customers, and reputations.

Azure Sphere consists of the following components:

• Azure Sphere–certified chips from hardware partners include built-in Microsoft security technology to provide connectivity and a dependable hardware root of trust.


• Azure Sphere OS adds layers of protection and ongoing security updates to create a trustworthy platform for new IoT experiences.


• Azure Sphere Security Service brokers trust for device-to-cloud communication, detects threats, and renews device security.

Together these components implement The Seven Properties of Highly Secure Devices .

Azure Sphere MediaTek MT3620 MCU architecture

The first Azure Sphere certified MCU is the MediaTek MT3620 microcontroller unit (MCU). This MCU which has three developer-accessible cores, a Cortex-A7 that runs a hardened Linux kernel, and two Cortex-M4 cores.

• The Cortex-A7 core runs less timing-sensitive tasks such as setting up network connections, negotiating security, refreshing certificates, updating the device OS and applications. You can run a custom application on this core and send and receive messages to predefined network endpoints including cloud gateways such as Azure IoT Hub.


• On the Cortex M4 cores you can run bare-metal code, or applications running on a real-time operating system such as Azure RTOS or FreeRTOS.

What is an RTOS (Real-Time Operating System)

A system is said to be real-time if the total correctness of an operation depends not only upon its logical correctness, but also upon the time in which it is performed Link to Wikipedia Article.

A Real-Time Operating System is system software that provides services and manages processor resources for applications. These resources include processor cycles, memory, peripherals, and interrupts. The main purpose of a real-time Operating System is to allocate processing time among various duties the embedded software must perform.

Why run real-time Azure RTOS applications on Azure Sphere

The main reasons to run real-time applications on an Azure Sphere include:

1. Your application requires precise or deterministic timing that cannot be guaranteed on the Cortex-A7 Linux kernel core where it would have to compete for resources with other services.


2. You are migrating existing Cortex M4 code to an Azure Sphere.


3. Running your application across multiple cores to take advantage of all the memory and processing resources on the Azure Sphere.


4. Dedicating a core to running compute intensive applications like machine learning models

Intercore communications

For security reasons, applications running on the Cortex M4 cores cannot directly access network endpoints. Applications can communicate with applications running on other cores through a secure mailbox mechanism. As an added layer of security, applications can only communicate across cores with applications they have been partnered with at development time.

Solution architecture example

This environment monitoring solution shows how you can integrate a real-time Azure RTOS application with Azure Sphere and IoT Central.

The solution architecture is as follows:

1. The Azure RTOS real-time environment sensor thread runs every 2 seconds. The thread stores in memory the latest environment temperature, humidity, and pressure data.


2. The high-level Azure Sphere applications requests environment data from the real-time Azure RTOS application.


3. The Azure RTOS real-time ‘environment service thread’ responds with the latest environment data.


4. The high-level application serializes the environment data as JSON and sends the telemetry message to IoT Central.


5. The IoT Central user can also set the desired temperature for the room by setting a property in IoT Central.


6. The Azure Sphere then sets the HVAC operating mode to meet the desired temperature.

Developer Experience

You can develop Azure Sphere and Azure RTOS applications on Linux and Windows. On Windows, you can develop high-level and real-time applications with Visual Studio (free Community Edition or better), Visual Studio Code, or integrate with your existing toolchains. On Linux you can develop high-level and real-time applications with Visual Studio Code or integrate with your existing toolchains.

Next Steps

There are two Microsoft Learn modules which include hands-on labs you can download to start your Azure Sphere and Azure RTOS journey.

1. Develop secure IoT solutions for Azure Sphere, Azure RTOS and Azure IoT Central


2. Develop secure IoT Solutions for Azure Sphere, Azure RTOS and IoT Hub

Skip to unit 16 to learn more about real-time Azure RTOS running on an Azure Sphere.