Citus Tips for Postgres: How to alter distribution key, shard count, & more

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Citus is an extension to Postgres that lets you distribute your application’s workload across multiple nodes. Whether you are using Citus open source or using Citus as part of a managed Postgres service in the cloud, one of the first things you do when you start using Citus is to distribute your tables. While distributing your Postgres tables you need to decide on some properties such as distribution column, shard count, colocation. And even before you decide on your distribution column (sometimes called a distribution key, or a sharding key), when you create a Postgres table, your table is created with an access method.

Previously you had to decide on these table properties up front, and then you went with your decision. Or if you really wanted to change your decision, you needed to start over. The good news is that in Citus 10, we introduced 2 new user-defined functions (UDFs) to make it easier for you to make changes to your distributed Postgres tables.

Before Citus 9.5, if you wanted to change any of the properties of the distributed table, you would have to create a new table with the desired properties and move everything to this new table. But in Citus 9.5 we introduced a new function, undistribute_table. With the undistribute_table function you can convert your distributed Citus tables back to local Postgres tables and then distribute them again with the properties you wish. But undistributing and then distributing again is … 2 steps. In addition to the inconvenience of having to write 2 commands, undistributing and then distributing again has some more problems:

1. Moving the data of a big table can take a long time, undistribution and distribution both require to move all the data of the table. So, you must move the data twice, which is much longer.


2. Undistributing moves all the data of a table to the Citus coordinator node. If your coordinator node isn’t big enough, and coordinator nodes typically don’t have to be, you might not be able to fit the table into your coordinator node.

So, in Citus 10, we introduced 2 new functions to reduce the steps you need to make changes to your tables:

• alter_distributed_table


• alter_table_set_access_method

In this post you’ll find some tips about how to use the alter_distributed_table function to change the shard count, distribution column, and the colocation of a distributed Citus table. And we’ll show how to use the alter_table_set_access_method function to change, well, the access method. An important note: you may not ever need to change your Citus table properties. We just want you to know, if you ever do, you have the flexibility. And with these Citus tips, you will know how to make the changes.

Altering the properties of distributed Postgres tables in Citus

When you distribute a Postgres table with the create_distributed_table function, you must pick a distribution column and set the distribution_column parameter. During the distribution, Citus uses a configuration variable called shard_count for deciding the shard count of the table. You can also provide colocate_with parameter to pick a table to colocate with (or colocation will be done automatically, if possible).

However, after the distribution if you decide you need to have a different configuration, starting from Citus 10, you can use the alter_distributed_table function.

alter_distributed_table has three parameters you can change:

• distribution column


• shard count


• colocation properties

How to change the distribution column (aka the sharding key)

Citus divides your table into shards based on the distribution column you select while distributing. Picking the right distribution column is crucial for having a good distributed database experience. A good distribution column will help you parallelize your data and workload better by dividing your data evenly and keeping related data points close to each other. However, choosing the distribution column might be a bit tricky when you’re first getting started. Or perhaps later as you make changes in your application, you might need to pick another distribution column.

With the distribution_column parameter of the new alter_distributed_table function, Citus 10 gives you an easy way to change the distribution column.

Let’s say you have customers and orders that your customers make. You will create some Postgres tables like these:

CREATE TABLE customers (customer_id BIGINT, name TEXT, address TEXT);
CREATE TABLE orders (order_id BIGINT, customer_id BIGINT, products BIGINT[]);

When first distributing your Postgres tables with Citus, let’s say that you decided to distribute the customers table on customer_id and the orders table on order_id.

SELECT create_distributed_table ('customers', 'customer_id');
SELECT create_distributed_table ('orders', 'order_id');

Later you might realize distributing the orders table by the order_id column might not be the best idea. Even though order_id could be a good column to evenly distribute your data, it is not a good choice if you frequently need to join the orderstable with the customers table on the customer_id. When both tables are distributed by customer_id you can use colocated joins, which are very efficient compared to joins on other columns.

So, if you decide to change the distribution column of orders table into customer_id here is how you do it:

SELECT alter_distributed_table ('orders', distribution_column := 'customer_id');

Now the orders table is distributed by customer_id. So, the customers and the orders of the customers are in the same node and close to each other, and you can have fast joins and foreign keys that include the customer_id.

You can see the new distribution column on the citus_tables view:

SELECT distribution_column FROM citus_tables WHERE table_name::text = 'orders';

How to increase (or decrease) the shard count in Citus

Shard count of a distributed Citus table is the number of pieces the distributed table is divided into. Choosing the shard count is a balance between the flexibility of having more shards, and the overhead for query planning and execution across the shards. Like distribution column, the shard count is also set while distributing the table. If you want to pick a different shard count than the default for a table, during the distribution process you can use the citus.shard_count configuration variable, like this:

CREATE TABLE products (id BIGINT, name TEXT);
SET citus.shard_count TO 20;
SELECT create_distributed_table ('products', 'id');

After distributing your table, you might decide the shard count you set was not the best option. Or your first decision on the shard count might be good for a while but your application might grow in time, you might add new nodes to your Citus cluster, and you might need more shards. The alter_distributed_table function has you covered in the cases that you want to change the shard count too.

To change the shard count you just use the shard_count parameter:

SELECT alter_distributed_table ('products', shard_count := 30);

After the query above, your table will have 30 shards. You can see your table’s shard count on the citus_tables view:

SELECT shard_count FROM citus_tables WHERE table_name::text = 'products';

How to colocate with a different Citus distributed table

When two Postgres tables are colocated in Citus, the rows of the tables that have the same value in the distribution column will be on the same Citus node. Colocating the right tables will help you with better relational operations. Like the shard count and the distribution column, the colocation is also set while distributing your tables. You can use the colocate_with parameter to change the colocation.

SELECT alter_distributed_table ('products', colocate_with := 'customers');

Again, like the distribution column and shard count, you can find information about your tables’ colocation groups on the citus_tables view:

SELECT colocation_id FROM citus_tables WHERE table_name IN ('products', 'customers');

You can also use default and none keywords with colocate_with parameter to change the colocation group of the table to default, or to break any colocation your table has.

To colocate distributed Citus tables, the distributed tables need to have the same shard counts. But if the tables you want to colocate don’t have the same shard count, worry not, because alter_distributed_table will automatically understand this. Then your table’s shard count will also be updated to match the new colocation group’s shard count.

How to change more than one Citus table property at a time

Here is a tip! If you want to change multiple properties of your distributed Citus tables at the same time, you can simply use multiple parameters of the alter_distributed_table function.

For example, if you want to change both the shard count and the distribution column of a table here’s how you do it:

SELECT alter_distributed_table ('products', distribution_column := 'name', shard_count := 35);

How to alter the Citus colocation group

If your table is colocated with some other tables and you want to change the shard count of all of the tables to keep the colocation, you might be wondering if you have to alter them one by one… which is multiple steps.

Yes (you can see a pattern here) the Citus tip is that you can use the alter_distributed_table function to change the properties of all of the colocation group.

If you decide the change you make with the alter_distributed_table function needs to be done to all the tables that are colocated with the table you are changing, you can use the cascade_to_colocated parameter:

SET citus.shard_count TO 10;
SELECT create_distributed_table ('customers', 'customer_id');
SELECT create_distributed_table ('orders', 'customer_id', colocate_with := 'customers');
-- when you decide to change the shard count 
-- of all of the colocation group
SELECT alter_distributed_table ('customers', shard_count := 20, cascade_to_colocated := true);

You can see the updated shard count of both tables on the citus_tables view:

SELECT shard_count FROM citus_tables WHERE table_name IN ('customers', 'orders');

How to change your Postgres table’s access method in Citus

Another amazing feature introduced in Citus 10 is columnar storage. This Citus 10 columnar blog post walks you through how it works and how to use columnar tables (or partitions) with Citus—complete with a Quickstart. Oh, and Jeff made a short video demo about the new Citus 10 columnar functionality too—it’s worth the 13 minutes to watch IMHO.

With Citus columnar, you can optionally choose to store your tables grouped by columns—which gives you the benefits of compression, too. Of course, you don’t have to use the new columnar access method—the default access method is “heap” and if you don’t specify an access method, then your tables will be row-based tables (with the heap access method.)

It would not be fair to introduce this cool new Citus columnar access method without also giving you a way to convert your tables to columnar. So Citus 10 also introduced a way to change the access method of tables.

SELECT alter_table_set_access_method('orders', 'columnar');

You can use alter_table_set_access_method to convert your table to any other access method too, such as heap, Postgres’s default access method. Also, your table doesn’t even need to be a distributed Citus table. You can also use alter_table_set_access_method with Citus reference tables as well as regular Postgres tables. You can even change the access method of a Postgres partition with alter_table_set_access_method.

Under the hood: How do these new Citus functions work?

If you’ve read the blog post about undistribute_table, the function Citus 9.5 introduced for turning distributed Citus tables back to local Postgres tables, you mostly know how the alter_distributed_table and alter_table_set_access_method functions work. Because we use the same underlying methodology as the undistribute_table function. Well, we improved upon it.

The alter_distributed_table and alter_table_set_access_method functions:

1. Create a new table in the way you want (with the new shard count or access method etc.)


2. Move everything from your old table to the new table


3. Drop the old table and rename the new one

Dropping a table for the purpose of re-creating the same table with different properties is not a simple task. Dropping the table will also drop many things that depend on the table.

Just like the undistribute_table function, the alter_distributed_table and alter_table_set_access_method functions do a lot to preserve the properties of the table you didn’t want to change. The functions will handle indexes, sequences, views, constraints, table owner, partitions and more—just like undistribute_table.

alter_distributed_table and alter_table_set_access_method will also recreate the foreign keys on your tables whenever possible. For example, if you change the shard count of a table with the alter_distributed_table function and use cascade_to_colocated := true to change the shard count of all the colocated tables, then foreign keys within the colocation group and foreign keys from the distributed tables of the colocation group to Citus reference tables will be recreated.

Making it easier to experiment with Citus—and to adapt as your needs change

If you want to learn more about our previous work which we build on for alter_distributed_table and alter_table_set_access_method functions go check out our blog post on undistribute_table.

In Citus 10 we worked to give you more tools and more capabilities for making changes to your distributed database. When you’re just starting to use Citus, the new alter_distributed_table and alter_table_set_access_method functions—along with the undistribute_table function—are all here to help you experiment and find the database configuration that works the best for your application. And in the future, if and when your application evolves, these three Citus functions will be ready to help you evolve your Citus database, too.

Remove Deprecated User Accounts across multiple Subscriptions

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Deprecated accounts are the accounts that were once deployed to a subscription for some trial/pilot initiative or some other purpose and are not required anymore. The user accounts that have been blocked from signing in should be removed from subscriptions. These accounts can be targets for attackers finding ways to access your data without being noticed.
The Azure Security Center recommends identifying those accounts and removing any role assignments on them from the subscription, however, it could be tedious in the case of multiple subscriptions.

function Get-apiHeader{
[CmdletBinding()]
Param
(
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $TENANTID,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $ClientId,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $PasswordClient,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $resource
)
$tokenresult=Invoke-RestMethod -Uri https://login.microsoftonline.com/$TENANTID/oauth2/token?api-version=1.0 -Method Post -Body @{"grant_type" = "client_credentials"; "resource" = "https://$resource/"; "client_id" = "$ClientId"; "client_secret" = "$PasswordClient" }
$token=$tokenresult.access_token
$Header=@{
  'Authorization'="Bearer $token"
  'Host'="$resource"
  'Content-Type'='application/json'
  }
return $Header
}

$AzureApiheaders = Get-apiHeader -TENANTID $TenantId -ClientId $ClientId -PasswordClient $ClientSecret -resource "management.azure.com"

Step 4: Extracting csv file containing list of all deprecated accounts from Azure Resource Graph

Please refer: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/governance/resource-graph/first-query-portal.md

Azure Resource graph explorer: https://docs.microsoft.com/en-us/azure/governance/resource-graph/overview

Query:

securityresources
        | where type == "microsoft.security/assessments"
        | extend source = tostring(properties.resourceDetails.Source)
        | extend resourceId =
            trim(" ", tolower(tostring(case(source =~ "azure", properties.resourceDetails.Id,
            extract("^(.+)/providers/Microsoft.Security/assessments/.+$",1,id)))))
        | extend status = trim(" ", tostring(properties.status.code))
        | extend cause = trim(" ", tostring(properties.status.cause))
        | extend assessmentKey = tostring(name)
        | where assessmentKey == "00c6d40b-e990-6acf-d4f3-471e747a27c4"

Click on “Download as CSV” and store at location where removal of deprecated account script is present. Rename the file as “deprecatedaccountextract“

Set-Location $PSScriptRoot
$RootFolder = Split-Path $MyInvocation.MyCommand.Path
$ParameterCSVPath =$RootFolder + "deprecatedaccountextract.csv"
if(Test-Path -Path $ParameterCSVPath)                                                                          
{ 
$TableData = Import-Csv $ParameterCSVPath
}

foreach($Data in $TableData)
{
  #Get resourceID for specific resource 
  $resourceid=$Data.resourceId
  #Get control results column for specific resource
  $controlresult=$Data.properties
  $newresult=$controlresult | ConvertFrom-Json
  $ObjectIdList=$newresult.AdditionalData.deprecatedAccountsObjectIdList
  $regex="[^a-zA-Z0-9-]"
  $splitres=$ObjectIdList -split(',')
  $deprecatedObjectIds=$splitres -replace $regex
  foreach($objectId in $deprecatedObjectIds)
  {   
    #API to get role assignment details from Azure
    $resourceURL="https://management.azure.com$($resourceid)/providers/microsoft.authorization/roleassignments?api-version=2015-07-01"
    $resourcedetails=(Invoke-RestMethod  -Uri $resourceURL -Headers $AzureApiheaders -Method GET)
    if( $null -ne $resourcedetails )
    {        
     foreach($value in $resourcedetails.value)
     {
      if($value.properties.principalId -eq $objectId)
      {
       $roleassignmentid=$value.name
       $remidiateURL="https://management.azure.com$($resourceid)/providers/microsoft.authorization/roleassignments/$($roleassignmentid)?api-version=2015-07-01"
       Invoke-RestMethod  -Uri $remidiateURL -Headers $AzureApiheaders -Method DELETE
      }
     }
    }
    else
    {
     Write-Output "There are no Role Assignments in the subscription"
    }             
   }
}

References:

https://github.com/Azure/Azure-Security-Center/blob/master/Remediation%20scripts/Remove%20deprecated%20accounts%20from%20subscriptions/PowerShell/Remove-deprecated-accounts-from-subscriptions.ps1

https://docs.microsoft.com/en-us/azure/security-center/policy-reference#:~:text=Deprecated%20accounts%20are%20accounts%20that%20have%20been%20blocked%20from%20signing%20in.&text=Virtual%20machines%20without%20an%20enabled,Azure%20Security%20Center%20as%20recommendations.

Removing Deprecated User Accounts across multiple Subscriptions

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

Deprecated accounts are the accounts that were once deployed to a subscription for some trial/pilot initiative or some other purpose and are not required anymore. The user accounts that have been blocked from signing in should be removed from subscriptions. These accounts can be targets for attackers finding ways to access your data without being noticed.
The Azure Security Center recommends identifying those accounts and removing any role assignments on them from the subscription, however, it could be tedious in the case of multiple subscriptions.

function Get-apiHeader{
[CmdletBinding()]
Param
(
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $TENANTID,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $ClientId,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $PasswordClient,
 [Parameter(Mandatory=$true)]
 [System.String]
 [ValidateNotNullOrEmpty()]
 $resource
)
$tokenresult=Invoke-RestMethod -Uri https://login.microsoftonline.com/$TENANTID/oauth2/token?api-version=1.0 -Method Post -Body @{"grant_type" = "client_credentials"; "resource" = "https://$resource/"; "client_id" = "$ClientId"; "client_secret" = "$PasswordClient" }
$token=$tokenresult.access_token
$Header=@{
  'Authorization'="Bearer $token"
  'Host'="$resource"
  'Content-Type'='application/json'
  }
return $Header
}

$AzureApiheaders = Get-apiHeader -TENANTID $TenantId -ClientId $ClientId -PasswordClient $ClientSecret -resource "management.azure.com"

Step 4: Extracting csv file containing list of all deprecated accounts from Azure Resource Graph

Please refer: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/governance/resource-graph/first-query-portal.md

Azure Resource graph explorer: https://docs.microsoft.com/en-us/azure/governance/resource-graph/overview

Query:

securityresources
        | where type == "microsoft.security/assessments"
        | extend source = tostring(properties.resourceDetails.Source)
        | extend resourceId =
            trim(" ", tolower(tostring(case(source =~ "azure", properties.resourceDetails.Id,
            extract("^(.+)/providers/Microsoft.Security/assessments/.+$",1,id)))))
        | extend status = trim(" ", tostring(properties.status.code))
        | extend cause = trim(" ", tostring(properties.status.cause))
        | extend assessmentKey = tostring(name)
        | where assessmentKey == "00c6d40b-e990-6acf-d4f3-471e747a27c4"

Click on “Download as CSV” and store at location where removal of deprecated account script is present. Rename the file as “deprecatedaccountextract“

Set-Location $PSScriptRoot
$RootFolder = Split-Path $MyInvocation.MyCommand.Path
$ParameterCSVPath =$RootFolder + "deprecatedaccountextract.csv"
if(Test-Path -Path $ParameterCSVPath)                                                                          
{ 
$TableData = Import-Csv $ParameterCSVPath
}

foreach($Data in $TableData)
{
  #Get resourceID for specific resource 
  $resourceid=$Data.resourceId
  #Get control results column for specific resource
  $controlresult=$Data.properties
  $newresult=$controlresult | ConvertFrom-Json
  $ObjectIdList=$newresult.AdditionalData.deprecatedAccountsObjectIdList
  $regex="[^a-zA-Z0-9-]"
  $splitres=$ObjectIdList -split(',')
  $deprecatedObjectIds=$splitres -replace $regex
  foreach($objectId in $deprecatedObjectIds)
  {   
    #API to get role assignment details from Azure
    $resourceURL="https://management.azure.com$($resourceid)/providers/microsoft.authorization/roleassignments?api-version=2015-07-01"
    $resourcedetails=(Invoke-RestMethod  -Uri $resourceURL -Headers $AzureApiheaders -Method GET)
    if( $null -ne $resourcedetails )
    {        
     foreach($value in $resourcedetails.value)
     {
      if($value.properties.principalId -eq $objectId)
      {
       $roleassignmentid=$value.name
       $remidiateURL="https://management.azure.com$($resourceid)/providers/microsoft.authorization/roleassignments/$($roleassignmentid)?api-version=2015-07-01"
       Invoke-RestMethod  -Uri $remidiateURL -Headers $AzureApiheaders -Method DELETE
      }
     }
    }
    else
    {
     Write-Output "There are no Role Assignments in the subscription"
    }             
   }
}

References:

https://github.com/Azure/Azure-Security-Center/blob/master/Remediation%20scripts/Remove%20deprecated%20accounts%20from%20subscriptions/PowerShell/Remove-deprecated-accounts-from-subscriptions.ps1

https://docs.microsoft.com/en-us/azure/security-center/policy-reference#:~:text=Deprecated%20accounts%20are%20accounts%20that%20have%20been%20blocked%20from%20signing%20in.&text=Virtual%20machines%20without%20an%20enabled,Azure%20Security%20Center%20as%20recommendations.

Azure IoT Edge – A Year In Review

by Contributed | May 3, 2021 | Technology

This article is contributed. See the original author and article here.

This past March marked the one-year anniversary of the covid-19 pandemic. I found myself reflecting on the changes I’ve gone through, both personally and professionally. One of the bright spots was realizing how much progress our customers have made with Azure IoT Edge and the number of improvements our team has been able to add to the product. Let’s take a moment to celebrate some of the successes in a year that most of us would prefer never happened.

Product maturity

COVID has caused a noticeable change in the types of scenarios in which customers are using Azure IoT Edge; however, it hasn’t dampened the market’s embrace of our product. In the past year, we documented over 25 new case studies of customers using Azure IoT Edge in their digital transformation. While these case studies are a sampling of the many customers using Azure IoT Edge, they are a great example of how widely used our product is. There is representation from a gamut of industries including: automotive, banking, energy, farming, healthcare, HVAC, industrial automation, manufacturing, packaging, real estate, recycling, retail, and shipping.

Azure IoT Edge hit an important milestone in March with the 1.1.0 release. It is our first long term servicing (LTS) release and will only be serviced with fixes to critical security issues and regressions. All other bug fixes and new feature work goes into our rolling feature releases (1.2.0 and greater). The limited number of changes in makes 1.1.0 it our most stable release and ideal for extended periods of use in production.

Security

Notable security exploits and data breaches continued in the tech industry last year. Azure IoT is a leader in edge security; however, we realize that we cannot rest on our laurels. To that end, we’ve shipped features specifically intended to increase the security posture of Azure IoT Edge.

First, we’ve updated the runtime to acquire needed certificates from EST standard infrastructure. Some customers require that certs come from their on-prem PKI infrastructure and leveraging the EST standard allows the product to integrate with compatible infrastructure.

Simply allowing customers to create their own certs is not enough. These certs must be protected from tampering, or theft, once on the device. Azure IoT Edge now uses the EST standard to integrate with hardware security modules (eg. TPM) to protect these valuable certs.

Just because a device starts off secure does not mean it remains secure. New vulnerabilities are discovered daily. Updating devices is a critical capability in the quest to keep assets protected from the latest threats. IoT Edge already does a great job of allowing customers to update the workload running on a device; however, patching the OS or even updating native components of the runtime was a job left up to the user. No more! Device Update for IoT Hub is in public preview. Customers can update IoT Edge devices, use an IoT Edge device as an update cache for downstream devices, and all of this works in a nested hierarchy often found in ISA 95 networks.

Observability

The ability to remotely monitor devices, understand how they are operating, and proactively identify problems is critical for solutions running in production. The IoT Edge runtime is now instrumented to produce metrics like resource consumption, message throughput, and module uptime. There’s a private preview detailing how customers can harvest these metrics and send them to the cloud to create intuitive dashboards, and we’re already working on making this default functionality you get out of the box.

The support experience has also been improved. Once you realize there is an issue which needs more investigation, customers must collect logs from various components and upload them to the cloud. Now the runtime is smart enough to automatically collect the pertinent logs and upload them on the customer’s behalf via the support-bundle command.

Industrial

Over a year ago we undertook a feature set aimed at unblocking industrial customers. We’re now starting to see the benefits of that work. Many industrial customers have ISA 95 networks, a standard that creates a hierarchical topology where networks are layered on top of each other. Devices in one layer can only talk to devices in the previous or subsequent layer. This throws a wrench in IoT Edge solutions because the IoT Edge devices cannot send telemetry, receive updates, or do anything else that requires the internet unless they are in the topmost layer.

IoT Edge in a nested configuration allows customers to create a daisy chain of IoT Edge devices that traverses the network hierarchy. The devices in isolated network levels leverage the connection of their parent to eventually gain access to the internet.

Nesting of IoT Edge devices doesn’t only solve the industrial problem of ISA 95 networks, it’s useful for any customer who wants to run analytics in any type of hierarchy. For example, a customer in smart buildings could use nested edge for granular optimization of energy usage. An IoT Edge device could be used to run a machine learning model which optimizes energy usage for a single floor. This device could then supply data to a parent IoT Edge device which is running AI that optimizes energy usage for the entire building.

Edge analytics

Digital transformation has multiple phases: Connect & Monitor, Analyze & Improve, Transform & Expand. While many customers are still working through the first phase, there are others who have graduated to running analytics on the edge. Azure IoT Edge continues to invest in this area so that we have powerful features available when customers are ready to leverage them. For example, the LTS branch of IoT Edge supports native integration with Nvidia deepstream so that customers can create devices which score AI models on multiple video streams in real time!

The development story for edge analytics is also being moved forwards in leaps and bounds. Recently the Percept family of products was announced. These are best in class dev kits which are built on Azure IoT Edge.

MQTT

We have heard the desire for an MQTT broker that supports customer defined topics, pub/sub semantics, and existing MQTT devices. Azure IoT Hub and Azure IoT Edge are working together to provide this capability in the cloud and on the edge. The beginning of this functionality is available on the edge in a public preview.

Azure IoT Edge on Windows

Windows has a long history of security and best in class manageability. These are key reasons that some customers want to run Azure IoT Edge on a Windows host machine. Unfortunately, using these two products together create issues which are not present when running Azure IoT Edge on Linux. For example, the base image of Azure IoT Edge modules must match the Windows host OS image and RS5 was the only supported OS. Or, partners had to maintain two versions of their modules, both a Linux container and Windows container, to target all Azure IoT Edge devices.

Azure IoT Edge For Linux On Windows, a technology in public preview, fixes these issues. It transparently spins up a Linux VM and runs Azure IoT Edge in the context of this Linux VM. Since the VM is transparent, customers continue to manage the Windows host as they would any other Windows device. Partners writing modules for Azure IoT Edge now only have to create and maintain Linux container versions of their modules since Azure IoT Edge now runs in a Linux environment, regardless of the OS used by the host.

Supporting features

The very large investments detailed above should not overshadow a handful of smaller point improvements that add critical functionality to Azure IoT Edge solutions.

• Message priority ensures that high priority messages are synced to the cloud before lower priority messages after a period of no connectivity.


• Module boot order provides information to the runtime used to try and start modules in a particular order. Please note, that this order is not binding (a module which is started first may take a while to initialize and a subsequent module could begin running before it). Module boot order also does not apply to restarts of individual modules (if the first module crashes, all subsequent modules continue to run and are not restarted with the failed module).  


• Module twins now support arrays. This provides module authors to more data structures for syncing information in module twins and prevents them from (serializing/deserializing) arrays (into/out of) module twins.

The work produced by the IoT Edge team over the past year is nothing short of amazing. It’s even more impressive when one realizes that it was delivered under such stressful conditions. Lorenzo and I are truly grateful to lead such a talented and motivated team. Working with them has been a highlight in a year where so many people have struggled. We have many more great innovations coming this year, and we can’t wait to see the solutions our customers continue to build with Azure IoT Edge.