This article is contributed. See the original author and article here.
In response to the enthusiastic adoption of the eleven Synapse Database Templates for which we announced availability, in public preview, in November and December 2021, we’re pleased to announce today that we are making available, in public preview, additional Synapse Database Templates for four more industries:
Automotive Industries
Genomics
Manufacturing
Pharmaceuticals
The Synapse Database Template for Automotive Industries is a comprehensive data model that addresses the typical data requirements of organizations engaged in manufacturing automobiles, heavy vehicles, tires, and other automotive components.
The Synapse Database Template for Genomics is a comprehensive data model that addresses the typical data requirements of organizations engaged in acquiring and analyzing genomic data about human beings or other species.
The Synapse Database Template for Manufacturing is a comprehensive data model that addresses the typical data requirements of organizations engaged in either discrete or continuous manufacturing of a wide range of products. Portions of this database template are also included as part of the previously announced Consumer Goods database template and in the Automotive Industries and Pharmaceuticals database templates being announced today so companies in those industries will likely prefer to use those database templates rather than the general purpose Manufacturing industry database template as they contain additional content specific to those industries in addition to the content contained in the general Manufacturing database template.
The Synapse Database Template for Pharmaceuticals is a comprehensive data model that addresses the typical data requirements of organizations engaged in creating, manufacturing, and marketing pharmaceutical and bio-pharmaceutical products and medical devices.
Most of the Synapse database templates contain many different business areas that together comprise each of these very large industry-specific data models. For example, in addition to business areas such as Product, Inventory, and Human Resources, there is also an Emissions business area which provides support for data used to report greenhouse gas emissions (including scope 1, scope 2, and scope 3 emissions), an area of significant focus for many of our large customers.
These 4 additional database templates can be accessed now in Azure Synapse either in the gallery or by creating a new lake database from the Data tab and selecting + Table and then From template.
This article is contributed. See the original author and article here.
This is the next segment of our blog series highlighting Microsoft Learn Student Ambassadors who achieved the Gold milestone and have recently graduated from university. Each blog in the series features a different student and highlights their accomplishments, their experience with the Student Ambassadors community, and what they’re up to now.
Today we meet Rogerio Rodrigues who is from Brazil and recently graduated from the Universidade Federal de São Paulo with a master’s degree in computer science.
Responses have been edited for clarity and length.
When you joined the Student Ambassador community in 2012 as an undergraduate student at Universidade Federal de São Paulo, did you have specific goals you wanted to reach?
I had some goals like learning to speak in public because I was very shy but wanted to contribute to public events in the Student Ambassador program. I also wanted to go deeper into Microsoft technologies like Azure, DevOps, and Mobile Development with Xamarin. I had a dream to work at Microsoft as an employee. The community helped me a lot with these goals. I learned public speaking and became an MBA technical training teacher. I learned a lot about Microsoft technologies, received recognition in the technical community, and became a Microsoft Most Valuable Professional (MVP). I am pleased to have been part of the community, which undoubtedly contributed the most to my professional career.
As a Student Ambassador, what were the accomplishments that you’re the proudest of and why?
In 2018 and 2019, together with other Student Ambassadors and Microsoft MVPs , we created an initiative called Natal Nerd where we collected gifts, toys, and cash as donations to help orphans and at-risk children for Christmas. Several Student Ambassador events and lectures about Microsoft technologies were held at universities with companies in São Paulo, and the entry fee was a toy to donate to Natal Nerd. We presented at 23 universities in the country and reached more than 2,500 students and professors, disseminating and sharing knowledge about Microsoft technologies and the Imagine Cup. We were able to collect gifts to improve Christmas for more than 800 children.
In 2019, another Student Ambassador and I delivered 4 talks at 11 technical events at the Microsoft Student Partners Tech Days at eight universities in Brazil with the help of MVPs [Editor’s note: Microsoft Student Partners is the former name of the Microsoft Learn Student Ambassadors program]. We delivered another four talks in countries such as Peru, Nicaragua, Colombia, and Mexico with the help of other Student Ambassadors.
In 2019 and 2020, I was chosen to go to the Microsoft Student Partner Summit and the Imagine Cup Final because of my contributions. The events in 2019 were amazing – opportunities that I will never forget because they was very valuable to me. Even though the events in 2020 were virtual, I was very happy to have received the recognition.
My dream was to become a Microsoft employee. I had already tried twice and didn’t pass the interviews, but last year I became a Customer Engineer at Microsoft and was assigned to serve large public sector banking clients. I’ve also started my part-time doctorate program in computer science with a focus on environmental science and weather satellites. After completing my doctorate, I plan to do a post-doc in my line of research, continue to teach at universities, and contribute to science.
If you could redo your time as a Student Ambassador, is there anything you would have done differently?
The only thing I would have done differently would be to study more English right away when I joined in order to communicate more easily with other Student Ambassadors around the world. Learning to speak English better made a big difference for me in 2019 when I was at the Student Partner Summit for the first time because in addition to learning new technical content, I met many other Student Ambassadors from other cultures and the amazing Student Ambassador Program Team.
If you were to describe the community to a student who is interested in joining, what would you say about it to convince him or her to join?
If you desire to be a unique professional, you should apply to the Student Ambassadors program. In addition to receiving special treatment in the community, you will have access to a lot of technical information, you will have the opportunity to network with people from all over the planet, and you will have contact with Microsoft professionals where you can find excellent opportunities.
Furthermore, the community helps you have a much broader view of technology than just Microsoft technologies, as you will have contact with different areas and the opportunity to work on fantastic projects.
What advice would you give to new Student Ambassadors?
Actively participate in the community. Join other Student Ambassadors in Leagues or groups to create initiatives. It’s fantastic, and you learn a lot. Make contact with other Student Ambassadors as you will experience different cultures, practice languages, learn more technology, and perhaps share your knowledge with other Student Ambassadors. Actively contribute to your local community and promote content at your university. These actions will help you to secure further knowledge and develop new skills. The community provides a sea of opportunities.
What is your motto in life, your guiding principle?
My motto in life is “Be different. Help lift but not drop”. Helping those we can help will bring us great experiences, and we will be doing our part in making the world better.
What is one random fact about you that few people are aware of?
I’m a big fan of Star Wars. I have a big tattoo on my back that’s the design of an X-Wing spaceship with the 3 characters I like–Luke, Leia, and Rey Skywalker.
This article is contributed. See the original author and article here.
By Erin Boris, Idan Basre and Yoann Mallet
One of the latest app connectors to be added to Microsoft Defender for Cloud Apps is for Smartsheet.
As we have in the past for other protected apps, we would like to share a few examples of how to use it to secure your Smartsheet deployment.
Why Connect Smartsheet?
As with other apps, connecting Smartsheet to Defender for Cloud Apps allows you to leverage some of the built-in features of your favorite CASB, such as:
Benefit
Description
Policy or template
Threat and Anomaly Detection
Detect cloud threats, compromised accounts, and malicious insiders
The following built-in Threat detection policies automatically apply when Defender for Cloud Apps is connected to Smartsheet:
Use the audit trail of activities for forensic investigations
Leverage Advanced Hunting Queries as part of the Microsoft 365 Defender Portal to audit the use of Smartsheet and create policies
How to connect Smartsheet?
Let’s start with connecting Smartsheet.
Detailed instructions are available here, and if you prefer our video, check it out below:
Leverage Microsoft Defender for Cloud Apps with Smartsheet
The best way to get quick value from Defender for Cloud Apps when protecting Smartsheet, is to use the Advanced Hunting feature in the Microsoft 365 Defender portal.
If you have not used that portal just yet, simply visit http://security.microsoft.com and you will be redirected to it.
Once there browse to advanced hunting. This will enable you to write your own KQL queries to gather relevant information from the Defender for Cloud Apps activity logs.
Here are two relevant examples:
Scenario 1: New user invited, has an external email address
Having a new user invited to Smartsheet with an external email address may be suspicious. In order to detect this, you can leverage the following KQL Query:
CloudAppEvents
| where Application == "Smartsheet"
|extend a=parse_json(RawEventData).additionalDetails
|extend email=a.emailAddress
| where ActionType == "User Accept Invite"
and email !contains "contoso.com"
| project Timestamp,Application, AccountDisplayName, ActionType,email
Below is an example of the result of that query in Advanced Hunting:
This can potentially help you detect any attempt to exfiltrate data or identify a malicious user enabling an external account.
Another option to view all accounts with external email addresses that currently have access to Smartsheet, is from the API connector properties in Defender for Cloud Apps.
A filter can be used to see all external accounts that have accepted an invitation.
Scenario 2: File sent as attachment to external email address
When files are sent as an attachment from Smartsheet directly, to an external email address, this can be a sign of data exfiltration.
In order to identify such activities, you can leverage the following KQL Query:
CloudAppEvents
| where Application == "Smartsheet"
|extend a=parse_json(RawEventData).additionalDetails
|extend Recipient=a.recipientEmail
| where ActionType =="Sheet Send As Attachment"
and Recipient !contains "contoso.com"
| project Timestamp,Application, AccountDisplayName, ActionType,Recipient
Creating alerts from these queries
In addition to detecting these actions, Microsoft 365 Defender also allows you to create your own custom detections, and identify when these occur in near real-time, as described here.
As you can imagine, when using such powerful queries, the sky is the limit! Feel free to share any relevant KQL query you have identified for Smartsheet in the comments below.
Resources:
For more information about the features discussed in this article, please read:
We welcome your feedback or relevant use cases and requirements for this pillar of Cloud App Security by emailing CASFeedback@microsoft.com and mentioning the area or pillar in Cloud App Security you wish to discuss.
Learn more
For further information on how your organization can benefit from Microsoft Cloud App Security, connect with us at the links below:
This article is contributed. See the original author and article here.
The two billion frontline workers, which represent 80 percent of the world’s workforce, have weathered risk, exhaustion, and ongoing disruption throughout the pandemic. From shutdowns to new safety protocols to increased workloads and dwindling inventory, the frontline is constantly facing challenges. Today we are releasing a set of technology innovations and partnerships to reduce stress on the frontline and empower the way they work and interact.
This article is contributed. See the original author and article here.
I have been on a journey to explore Azure IoT and push the thousands of events that flow through my local MQTT broker (Mosquitto) into Azure IoT.
After my last post in using the Azure IoT SDK for Python conjunction with Paho MQTT I thought my work here was complete. But I have just recently been made aware that there is native support for various Arduino devices by Microsoft and Espressif. How awesome is that!
Before you get too excited given the requirements of such libraries, this is not going to work on your Arduino Uno, Arduino Mega 2560 and so on. Support for Azure IoT Hub is (for now) reserved for the newer generation of boards from Espressif (ESP32, ESP8266) and the Realtek Ameba D. These boards can contain megabytes, not kilobytes of RAM, multi-core CPU’s and are able to load in a TCP/IP stack, MQTT and so on.
If there is a theme for my house, it is bookended with reliability, and with that, it’s time to put my rack-mounted Raspberry Pi away and adopt a microcontroller. A Raspberry Pi, as great as it is, is an SBC (Single Board Computer) that needs to be updated, watered and fed. It uses a file system, a flash memory subsystem. How does this bode for reliability, and have you ever had a corrupt file system on a microcontroller?
Like any good, opinionated architect, I would urge you to stop, put away your Raspberry Pi’s and take a different approach: a microcontroller.
Today, I leverage around 30 outputs on an Arduino Mega 2560 with an Ethernet and PoE shield using MQTT (The pub/sub client library) but it’s time to modernise, and given my love of ESP devices with Tasmota, I decided to purchase an ESP32 for this very task.In this post I will illustrate how to build a bridge from Mosquitto MQTT into Azure IoT Hub using this ESP32 device.
I covered in a prior post why I am going down this path of publishing telemetry to Azure IoT Hub, along with the several ways I have illustrated how one can go about achieving this goal. From direct connection to Azure IoT Hub (via MQTT and SAS tokens) through to Azure IoT Edge running locally with MQTT and finally the SDK’s.I have been able to achieve my goals with varying levels of success but have a few concerns on the approaches I have tried thus far.
Direct-Connection to Azure IoT Hub introduces latency to the cloud.
Authentication, from SAS tokens to X509 certificates: it’s not anonymous and some of my tiny devices (Tasmota) dont bode well.
Topic structure: it is defined (devices/{DeviceID}/messsages/events/) and not free form. It means reconfiguration, which isn’t hard, but a lot of friction.
Reliability: all solutions thus far have relied on a OS which require patching, updating and are even whilst small an administrative burden.
My goals for building a solution
No reconfiguration of any of my MQTT devices (Home Assistant, PLC, Arduino Mega 2560, ~75 Tasmota devices).
Bridge my existing MQTT broker (Mosquitto) in to Azure IoT.
Run on microcontroller, as I want to be reliable.
Pretty lofty goals, you may even say I am being lazy, but the reality is I want a low friction away to derive operational intelligence from the many thousands of events each day (read below, it’s over 10K per day!)
What we are going to build
To overcome, the limitations described above we are going to use an ESP32 microcontroller with C++ code with a libraries. Just incase you are not familar, let me introduce you to the ESP32.
ESP32
Where do I start? What is not love about this SOC? The ESP32 is a modern, powerful Arduino compliant microcontroller that power many devices from my irrigation controller (Opensprinkler) through to my kids learning robot (MBot) they are either using an ESP32 or an older derivative such as an ESP8266. Today I am using this as a software bridge but there is a plethora of I/O and support for PWM, I2c and more which make them a versatile all rounder.
The ESP32 is a series of low-cost, low-power system on a chipmicrocontrollers with integrated Wi-Fi and dual-mode Bluetooth. The ESP32 series employs either a Tensilica Xtensa LX6 microprocessor in both dual-core and single-core variations, Xtensa LX7 dual-core microprocessor or a single-coreRISC-V microprocessor and includes built-in antenna switches, RF balun, power amplifier, low-noise receive amplifier, filters, and power-management modules. ESP32 is created and developed by Espressif Systems, a Shanghai-based Chinese company, and is manufactured by TSMC using their 40 nm process. It is a successor to the ESP8266 microcontroller.
See the steps below as I tease out this solution or my GitHub repo for the full Arduino sketch. To give you a better understanding on how this works I will break it down in to the logical steps below required to receive messages from Mosquitto over MQTT using
‘PubSubClient’ and to then re-publish them in to Azure IoT Hub using the ‘Esp32MQTTClient’.
Step 1 – Arduino IDE – Add ESP32 to the Board Manager
The Arduino IDE does not know about the ESP32 so the very first step we need to do leverage the the Arduino IDE’s ‘Board Manager’ capability to provide support for the ESP32. In the Arduino IDE, open ‘Preferences’ and enter in one of the following URL’s
Open ‘Boards Manager’ from ‘Tools’ > ‘Board’ menu. Search for an install ‘ESP32’. Select your specific ESP32 board from the menu post installation.
Restart the Arduino IDE.
Step 2 – We Need A Library – PubSubClient
Whilst we now have support for the ESP32, we need to add a library that will allow us to subscribe to and receive MQTT messages from our Mosquitto broker. For this very purpose we need a MQTT library. There are many but I have used ‘PubSubClient’ in the past on other projects without any issues. To install, ‘Tools’ > ‘Manage Libraries’ > ‘PubSubClient’
Step 3 – Author Some Code (Libraries and Variables)
After validating your board is working (I would suggest uploading a Blink sketch) we can start coding. This example is based off the ‘Examples > ESP32 Azure IoT Arduino > Simple MQTT’;
We need to include some libraries, we will be using the Wi-Fi (for connectivity), PubSubClient (for Mosquitto MQTT) and the ESP32MQTTClient (for Azure IoT Hub).
Regarding Azure IoT Hub you will need to define your connection string. This post does not cover creating an IoT Hub or creating a device and assumed you have created this prior. See Use the Azure portal to create an IoT Hub | Microsoft Docs for more information on creating an Azure IoT Hub, adding a device and obtaining a device connection string.
Step 4 – Author Some Code (Setup Function: Connect to Wi-Fi , Azure and Mosquitto MQTT)
Our ‘setup’ function will establish connection to our LAN via Wi-Fi and then connect in to Azure where as the ‘MQTTConnect’ function not only connects to our local MQTT broker, but it defines the MQTT topics to subscribe to. You can subscribe to multiple MQTT topics by having multiple subscribe lines. You can also use MQTT wildcard filters to match events using fewer subscriptions.
Plus sign (+): It is a single level wildcard that matches any name for a specific topic level. We can use this wildcard instead of specifying a name for any topic level in the topic filter.
Hash (#): It is a multi level wildcard that we can use only at the end of the topic filter, as the last level and matches any topic whose first levels are the same as the topic levels specified at the left-hand side of the # symbol.
The serial monitor is handy in debugging any issues either with Wi-Fi or connecting in to Azure IoT Hub.
client.on_message = on_message
void setup() {
//Set baud rate
Serial.begin(115200);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("ESP32 : Connecting to WiFi...");
}
Serial.println("ESP32 : WiFi connected");
Serial.println("ESP32 : IP address: ");
Serial.println(WiFi.localIP());
//Set MQTT details
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
//Connect to Azure IOT
if (!Esp32MQTTClient_Init((const uint8_t*)connectionString))
{
hasIoTHub = false;
Serial.println("Azure IoT Hub : Initializing IoT hub failed.");
return;
}
hasIoTHub = true;
}
void MQTTConnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("MQTT : Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP32Client")) {
Serial.println("MQTT : Connected");
// Once connected, publish an announcement...
client.publish("stat/ESP32/IP_Address","Your IP Address");
//Subscribe to topics, one topic per line.
client.subscribe("stat/+/POWER");
} else {
Serial.print("MQTT : Failed to connect to MQTT , rc=");
Serial.print(client.state());
Serial.println("MQTT : Trying again to connect to MQTT in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
Step 4 – Author Some Code (MQTT Call Back & Publish To Azure)
After the setup functions we now need to create a function that will listen for incoming MQTT messages that match our subscription (callback), extract the topic and payload before massaging this data and sending to Azure via another function (AzureIoTHub).
void callback(char* topic, byte* payload, unsigned int length) {
MQTTTopic = String(topic);
MQTTPayload = "";
for (int i = 0; i < length; i++) {
// Serial.print((char)payload[i]); - Use for debugging
MQTTPayload = String(MQTTPayload + (char)payload[i]);
}
}
void AzureIoTHub() {
if (hasIoTHub)
{
String tempString;
tempString = "{" + MQTTTopic + ":" + MQTTPayload + "}";
if (Esp32MQTTClient_SendEvent(tempString.c_str()))
{
Serial.println("Azure IoT Hub : Sending data to Azure IoT Hub succeed");
}
else
{
Serial.println("Azure IoT Hub : Failure...");
}
MQTTPayload = "";
MQTTTopic = "";
}
}
Step 5 – Author Some Code (Our Main Loop)
The main loop is leveraging all of these functions and its logic can be best sumarised in to a few points. Connect to MQTT if there is no connection
If there is a MQTT Topic/Message which was decoded via our ‘callback’ function send this to Azure IoT Hub and re-connect if there is no connection.
void loop() {
//Connect to MQTT and reconnect if connection drops
if (!client.connected()) {
MQTTConnect();
}
//Respond to messages received
if (MQTTTopic != "") {
Serial.println("MQTT : Topic is [" + MQTTTopic +"]");
Serial.println("MQTT : Payload is [" + MQTTPayload + "]");
AzureIoTHub();
}
client.loop();
}
Pulling It All Together
Here is a complete copy of the above, plus a bit more. You could cut and paste the below or clone my GitHub repository.
#include <WiFi.h>
#include <PubSubClient.h>
#include "Esp32MQTTClient.h"
const char* ssid = "****";
const char* password = "****";
const char* mqttServer = "****";
const int mqttPort = 1883;
String MQTTTopic;
String MQTTPayload;
//Azure IOT Hub Setup
static const char* connectionString = "****";
static bool hasIoTHub = false;
WiFiClient espClient;
PubSubClient client(espClient);
void callback(char* topic, byte* payload, unsigned int length) {
MQTTTopic = String(topic);
MQTTPayload = "";
for (int i = 0; i < length; i++) {
// Serial.print((char)payload[i]); - Use for debugging
MQTTPayload = String(MQTTPayload + (char)payload[i]);
}
}
void MQTTConnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("MQTT : Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP32Client")) {
Serial.println("MQTT : Connected");
// Once connected, publish an announcement...
client.publish("stat/ESP32/IP_Address","Your IP Address");
//Subscribe to topics, one topic per line.
client.subscribe("stat/+/POWER");
} else {
Serial.print("MQTT : Failed to connect to MQTT , rc=");
Serial.print(client.state());
Serial.println("MQTT : Trying again to connect to MQTT in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
void setup() {
//Set baud rate
Serial.begin(115200);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("ESP32 : Connecting to WiFi...");
}
Serial.println("ESP32 : WiFi connected");
Serial.println("ESP32 : IP address: ");
Serial.println(WiFi.localIP());
//Set MQTT details
client.setServer(mqttServer, mqttPort);
client.setCallback(callback);
//Connect to Azure IOT
if (!Esp32MQTTClient_Init((const uint8_t*)connectionString))
{
hasIoTHub = false;
Serial.println("Azure IoT Hub : Initializing IoT hub failed.");
return;
}
hasIoTHub = true;
}
void loop() {
//Connect to MQTT and reconnect if connection drops
if (!client.connected()) {
MQTTConnect();
}
//Respond to messages received
if (MQTTTopic != "") {
Serial.println("MQTT : Topic is [" + MQTTTopic +"]");
Serial.println("MQTT : Payload is [" + MQTTPayload + "]");
AzureIoTHub();
}
client.loop();
}
void AzureIoTHub() {
if (hasIoTHub)
{
String tempString;
tempString = "{" + MQTTTopic + ":" + MQTTPayload + "}";
if (Esp32MQTTClient_SendEvent(tempString.c_str()))
{
Serial.println("Azure IoT Hub : Sending data to Azure IoT Hub succeed");
}
else
{
Serial.println("Azure IoT Hub : Failure...");
}
MQTTPayload = "";
MQTTTopic = "";
}
}
Seeing This In Action
Lets drop to a video to see this in working end-to-end, to validate messages are flowing in to Azure IoT Hub I can use the Azure CLI (AZ-CLI) to monitor the output coupled with the Arduino Serial monitor.
After 24 hours of running, we can see I have published 10.52K of messages in to Azure IoT Hub and there are certain ebbs and flows that occur in my house.
24 hour period of messages flowing in to Azure IoT Hub
Conclusion
There are many ways to skin this code cat. My requirements was to publish messages in to Azure and we have been able to achieve this via different ways (I am sure there is more). Automation is a journey, which path will you take?
We illustrated a transparent side-car approach that will listen to an existing broker, on topics you desire and push these in to Azure IoT, all without making any configuration changes (the most important thing for my implementation). This method runs on a microcontroller, consumes less than 5w of power and just works.
Are there any draw backs? Sure there are. Right now this is one way in direction (simplex) and allows me to push messages in to Azure IoT but not receive messages back.
Personally, I like this approach, it combines the elegance of a SDK as it’s my code and couples the reliability of a microcontroller. It’s my code, my choices on what I do, but I do understand this is not for everyone. We now have my messages, my events, in Azure and it’s time to make some friends and learn how to derive operational intelligence from visualizations through to machine learning and beyond.
Recent Comments