Microsoft Archives - Page 180 of 1135 - Dr. Ware Technology Services

New transactable offers from Dace IT and PhakamoTech in Azure Marketplace

by Contributed | Jul 28, 2022 | Technology

This article is contributed. See the original author and article here.

Microsoft partners like Dace IT and PhakamoTech deliver transact-capable offers, which allow you to purchase directly from Azure Marketplace. Learn about these offers below:

Intelligent Traffic Management 2022: Detect and track bikes, vehicles, and pedestrians, as well as collisions and near-misses with Intelligent Traffic Management from Dace IT. This data can be used to adjust traffic lights for traffic flow optimization and automatically notify emergency services.

Managed Cybersecurity Operation Center Service: This multi-tenant 24/7 cybersecurity operations managed service from PhakamoTech will enhance your security investments using Microsoft Azure Sentinel, Microsoft Defender for Cloud, and more. This service centralizes visibility for better threat detection, response, and compliance setting.

Single to Flexible server PostgreSQL migration tool is now in Preview!

by Contributed | Jul 27, 2022 | Technology

This article is contributed. See the original author and article here.

Many enterprises deploy applications using Azure Database for PostgreSQL Single Server, a fully managed Postgres database service, that is best suited for minimal customizations of your database. In November 2021, we announced that the next generation of the Azure Database for PostgreSQL, Flexible Server, was generally available. Since then, customers have been using methods like dump/restore, Azure Database Migration Service (DMS), and custom scripts to migrate their databases from Single to Flexible Server.

If you are a user of Azure Database for PostgreSQL Single Server, and looking to migrate to Postgres Flexible Server, At Microsoft, our mission is to empower every person and organization on the planet to achieve more. It’s this mission that drives our firm commitment to collaborate and contribute to the Postgres community and invest in bringing the best migration experience to all our Postgres users.

Why migrate to Azure Database for PostgreSQL Flexible Server?

If you are not familiar with Flexible server, it provides you with the simplicity of a managed database service together with maximum flexibility over your database, and built-in cost-optimization controls. Azure Database for PostgreSQL Flexible Server is the next generation Postgres service in Azure and offers a robust value proposition and benefits including:

Infrastructure: Linux-based VMs, premium managed disks, zone-redundant backups

Cost optimization/Dev friendly: Burstable SKUs, start/stop features, default deployment choices

Additional Improvements over Single Server: Zone-redundant High Availability (HA), support for newer PG versions, custom maintenance windows, connection pooling with PgBouncer, etc.

Learn more about why Flexible Server is the top destination for Azure Database for PostgreSQL

Top 10 reasons to choose Flexible Server

What is exciting in Flexible Server

Modernize your applications using Flexible Server

Why do you need the Single to Flexible Server PostgreSQL migration tool?

Now, let’s go over some of the nuances of migrating data from Single to Flexible Server.

Single Server and Flexible Server run on different OS platforms (Windows vs Linux) with different physical layouts (Basic/ 4TB/16TB storage Vs Managed disk), and different collation. Flexible Server supports PostgreSQL DB versions 11 and above. If you are using PG 10 and below on Single Server (those are retired, by the way), you must make sure your application is compatible with higher versions. These challenges prohibit us from making a simple physical copy of the data. Only a logical data copy, such as dump/restore (offline mode), or logical decoding (online mode) can be performed.

While the offline mode of migration is simple, many customers prefer an online migration experience that keeps your source PostgreSQL server up and operational until the cutover, however, there is a lengthy checklist of things to be aware of during this process. These steps include

Setting the source’s WAL_LEVEL to LOGICAL

Schema copy

Creation of databases at the target,

Disabling foreign keys and triggers at the target.

Enabling them post migration, and so on.

Here is where the Single to Flexible migration tool comes into help to make the migration experience simpler for you.

What is Single to Flexible Server PostgreSQL migration tool?

This migration tool is tailormade to alleviate the pain of migrating the PostgreSQL schema and data from Single to Flexible Server. This tool offers a seamless migration experience, while abstracting the migration complexities under the hood. This tool is targeted for less than 1 TiB database size. This migration tool automates the majority of the migration steps, allowing you to focus on minor administration and pre-requisite tasks.

You can either use the Azure portal or Azure CLI to perform online or offline migrations.

Offline mode: This offers a simple database dump and restore of the data and schema and is best for smaller sized databases where downtime can be afforded.

Online mode: This mode of migration performs data dump and restore, and initiates change data capture using Postgres logical decoding and is best in scenarios where downtime cannot be afforded.

For a detailed comparison, see this documentation.

Figure 1: Block diagram to show the migration feature steps and processes

Create a Flexible PostgreSQL server (public or VNET)

Invoke migration from Azure Portal or Azure CLI and choose databases to migrate.

A Migration infrastructure is provisioned (DMS) on your behalf

Initiates migration
- (4a) Initial dump/restore (online & offline)
- (4b) streams the changes using logical decoding – CDC (online only)

You can then cutover if you are doing online migration. If you are using offline mode, then post restore, the cutover is automatically performed.

What are the steps involved for a successful migration?

1. Planning for Single to Flexible PostgreSQL server migration

This is a very critical step. Some of the steps in the planning phase include:

Getting the list of source Single servers, SKUs, storage, public/private access (Discovery)

Single server provides Basic, General purpose, and Memory optimized SKUs. Flexible Server offers Burstable SKUs, General purpose, and Memory optimized SKUs. While General purpose and Memory optimized SKUs can be migrated to the equivalent SKUs, for your Basic SKU, you can consider either Burstable SKU or General purpose SKU depending in your workload. See Flexible server compute & storage documentation for details.

Get the list of each database in the server, the size, and extensions usage. If your database sizes are larger than 1TiB, then we recommend you reach out to your account team or contact us at AskAzureDBforPostgreSQL@service.microsoft.com to help with your migration requirement..

Decide on the mode of migration. This may require batching of databases for each server.

You can also choose to do a different Database:Server layout compared to Single Server. For example, you can choose to consolidate databases on Flexible (or) you want to spread out databases across multiple Flexible servers.

Plan for the day/time for migrating your data to make sure your activity is reduced at the source server.

2. Migration pre-requisites

Once you have the plan in place, take care of few pre-requisites. For example

Provision Flexible Server in public/VNET with the desired compute tier.

Set the source Single Server WAL level to LOGICAL.

Create an Azure AD application and register with the source server, target server, and the Azure resource group.

3. Migrating your databases from Single to Flexible Server PostgreSQL

Now that you have taken care of pre-requisites, you can invoke migration using the Azure Portal or Azure CLI. Create one or more migration tasks using Azure portal or Azure CLI. High-level steps include

Select the source server

Choose up to 8 databases per migration task

Choose online or offline mode of migration

Select network to deploy the migration infrastructure (if using private network).

Create the migration

Following the invocation of the migration, the source and the target server and details are validated before initiating the migration infrastructure using Azure DMS, copying of schema, perform dump/restore steps, and if doing online then continue with CDC.

Verify data at the target

If doing online migration, when ready, perform cutover.

4. Post migration tasks

Once the data is available in the target Flexible PostgreSQL server, perform post-migration tasks including copying of roles, recreating large objects and copying of settings such as server parameters, firewall rules, monitoring alerts, and tags to the target Flexible server.

What are the limitations?

You can find a list of limitations in this documentation.

What do our early adopter customers say about their experience?

We’re thankful to our many customers who have evaluated the managed migration tooling and trusted us with migrating their business-critical applications to Azure Database for PostgreSQL Flexible Server. Customers appreciate the migration tool’s ease of use, features, and functionality to migrate from different Single Server configurations. Below are comments from a few of our customers who have migrated to the Flexible Server using the migration tool.

Allego, a leading European public EV charging network, continued to offer smart charging solutions for electric cars, motors, buses and trucks without interruption. Electric mobility increases the air quality of our cities and reduces noise pollution. To fast forward the transition towards sustainable mobility Allego believes that anyone with an electric vehicle should be able to charge whenever and wherever they need. That’s why we have partnered with Allego and are working towards providing simple, reliable and affordable charging solutions. Read more about the Allego story here.

“The Single to Flexible migration tool was critical for us to have a minimal downtime. While migrating 30 databases with 1.4TB of data across Postgres versions, we had both versions living side-by-side until cutover and with no impact to our business.”

Mr. Oliver Fourel, Head of EV Platform, Allego

Digitate is a leading software provider bringing agility, assurance, and resiliency to IT and business operations.

“We had a humongous task of upgrading 50+ PostgreSQL 9.6 of database size 100GB+ to a higher version in Flexible Postgres server. Single to Flexible Server Migration tool provided a very seamless approach for migration. This tool is easy to use with minimum downtime.”

Vittal Shirodkar, Azure specialist, Digitate

Why don’t you try migrating your Single server to Flexible PostgreSQL server?

If you haven’t explored Flexible server yet, you may want to start with Flexible Server docs, which provide a great place to roll up your sleeves. Also visit our website to learn more about our Azure Database for PostgreSQL managed service.

Check out the Single Server to Flexible server migration tool demo in the Azure Database for PostgreSQL migration on-demand webinar.

We look forward to helping you have a pleasant migration experience to Flexible Server using the migration feature. If you would like to reach out us about the migration experience or migrations to PostgreSQL in general, you can always reach out to our product team via email at Ask Azure DB for PostgreSQL.

3 New Data Transformation Functions in ADF

by Contributed | Jul 26, 2022 | Technology

This article is contributed. See the original author and article here.

Azure Data Factory and Azure Synapse Analytics Mapping Data Flows provides powerful, easy-to-use, data transformation at cloud scale. We’ve recently introduced 3 new data transformation functions to our rich data flow expression language: collectUnique(), substringIndex(), and topN().

collectUnique()
- Create a new collection of values into an array. ADF will automatically dedupe elements in the array for you.

substringIndex()
- Extracts the substring before n ccurrences of the delimiter.

topN()
- ADF will sort your data based on the column or expression that you provide and then return the top n results.

Find the documentation for all ADF data flow transformation functions at this link here.

Stream Microsoft Defender for IoT alerts to a 3rd party SIEM

by Contributed | Jul 25, 2022 | Technology

This article is contributed. See the original author and article here.

Overview

As more businesses convert OT systems to digital IT infrastructures, security operations center (SOC) teams and chief information security officers (CISOs) are increasingly responsible for handling threats from OT networks.

Defender for IoT’s built-in integration with Sentinel helps bridge the gap between IT & OT security challenge. Sentinel enables SOC teams to reduce the time taken to manage and resolve OT incidents efficiently by providing out-of-the-box capabilities to analyze OT security alerts, investigate multistage IT/OT attacks, utilize Azure Log Analytics for threat hunting, utilize threat intelligence, and automate incident response using SOAR playbooks.

Customer engagements have taught us that sometimes customers prefer to maintain their existing SIEM, alongside Microsoft Sentinel, or as a standalone SIEM.

In this blog, we’ll introduce a solution that sends Microsoft Defender for IoT alerts to an Event Hub that can be consumed by a 3^rd party SIEMs. You can use this solution with Splunk, QRadar, or any other SIEM that supports Event Hub ingestion.

Preparation and use

In this blog, we’ll use Splunk as our example.

The following describe the necessary preparation steps:

Connect your alerts from Defender for IoT to Microsoft Sentinel

Create an Azure Event Hub Namespace

Prepare Azure Sentinel to forward Incidents to Event Hub

Configure Splunk to consume Azure Sentinel Incidents from Azure Event Hub

1. Connect your alerts from Defender for IoT to Microsoft Sentinel

The first step is to enable the Defender for IoT data connector so that all Defender for IoT alerts are streamed into Microsoft Sentinel (a free process).

In Microsoft Sentinel, under Configuration, select Data Connectors and then locate Microsoft Defender for IoT data connector. Open the connector page, select the subscription whose alerts you want to stream into Microsoft Sentinel, and then select Connect.

For more information, see Connect your data from Defender for IoT to Microsoft Sentinel

2. Register an application in Azure AD

You’ll need Azure AD to be defined as a service principal for Splunk Add-on for Microsoft Cloud Services.

To register an app in Azure AD, open the Azure Portal and navigate to Azure Active Directory > App Registrations > New Registration. Fill the Name and click Register.

Click Certificates & secrets to create a secret for the Service Principle. Click New client secret and note its value.

To grant the required permissions to read data from the app, click API permissions > Add a permission and select Microsoft Graph > Application permissions > SecurityEvents.ReadWrite.All.

Ensure that the granted permission is approved by admin.

For the next step of setting up Splunk Add-on for Microsoft Cloud Services, note the following settings:
- The Azure AD Display Name
- The Azure AD Application ID
- The Azure AD Application Secret
- The Tenant ID

3. Create an Azure Event Hub Namespace

In the Azure Portal, navigate to Event Hubs > New to create a new Azure Event Hub Namespace. Define a Name, select the Pricing Tier and Throughput Units and click Review + Create.

Once the Azure Event Hub Namespace is created click Go to resource and click + Event Hubs to create an Azure Event Hub within the Azure Event Hub Namespace.

Define a Name for the Azure Event Hub, configure the Partition Count, Message Retention and click Review + Create.

Navigate to Access control (IAM) and Click + Add > Add role assignment to add the Azure AD Service Principle created before and delegate as Azure Event Hubs Data Receiver and click Save.

For the configuration of Splunk Add-on for Microsoft Cloud Services app, make a note of following settings:
- The Azure Event Hub Namespace Host Name
- The Azure Event Hub Name

4. Prepare Azure Sentinel to forward Incidents to Event Hub

To forward Microsoft Sentinel incidents or alerts to Azure Event Hub, you’ll need to define your Microsoft Sentinel workspace with a data export rule.

In the Azure Portal, navigate to Log Analytics > select the workspace name related to Microsoft Sentinel > Data Export > New export rule.

Name the rule, configure the Source as SecurityIncident and the Destination as Event Type utilizing the Event Hub Namespace and Event Hub Name configured previously. Click on Create.

5. Configure Splunk to consume Microsoft Sentinel Incidents from Azure Event Hub

For Microsoft Defender for IoT alerts to be ingested into Azure Event Hub, install the Splunk Add-on for Microsoft Cloud Services app.

For the installation, open the Splunk portal and navigate to Apps > Find More Apps. For the dashboard find the Splunk Add-on for Microsoft Cloud Services app and Install.

To add the Azure AD Service Principal, open the Splunk app and navigate to Azure App Account > Add. Use the details you’d noted earlier:

Define a Name for the Azure App Account

Add the Client ID, Client Secret, Tenant ID

Choose Azure Public Cloud as Account Class Type

Click Update to save and close the configuration.

Now navigate to Inputs within the Splunk Add-on for Microsoft Cloud Services app and select Azure Event Hub in Create New Input selection.

Define a Name for the Azure Event Hub as Input, select the Azure App Account created before, define the Event Hub Namespace (FQDN), Event Hub Name, let the other settings as default and click Update to save and close the configuration.

Once the ingestion is processed, you can query the data by using sourcetype=”mscs:azure:eventhub” in search field.

Disclaimer: The use of EventHub and Log Analytics export rule may incur an additional charge. For more information, see Event Hubs pricing and Log Data Export pricing

Speech Recognition for Singlish

by Contributed | Jul 23, 2022 | Technology

This article is contributed. See the original author and article here.

Mithun Prasad, PhD, Senior Data Scientist at Microsoft and Manprit Singh, CSA at Microsoft

Speech is an essential form of communication that generates a lot of data. As more systems provide a modal interface with speech, it becomes critical to be able to analyze human to computer interactions. Interesting market trends point that voice is the future of UI. This claim is further bolstered now with people looking to embrace contact less surfaces with the recent pandemic.

Interactions between agents and customers in a contact center remains dark data that is often untapped. We believe the ability to transcribe speech in the local dialects/slang should be in the midst of a call center advanced analytics road map such as the one proposed in this McKinsey recommendation. To enable this, we want to bring the best from the current speech transcription landscape, and present it in a coherent platform which businesses can leverage to get a head start on local speech to text adaptation use cases.

There is tremendous interest in Singapore to understand Singlish.

Singlish is a local form of English in Singapore that blends words borrowed from the cultural mix of communities.

An example of what Singlish looks like

A speech recognition system that could interpret and process the unique vocabulary used by Singaporeans (including Singlish and dialects) in daily conversations is very valuable. This automatic speech transcribing system could be deployed at various government agencies and companies to assist frontline officers in acquiring relevant and actionable information while they focus on interacting with customers or service users to address their queries and concerns.

Efforts are on to understand calls made to transcribe emergency calls at Singapore’s Civil Defence Force (SCDF) while AI Singapore has launched Speech Lab to channel efforts in this direction. Now, with the release of the IMDA National Speech Corpus, local AI developers now have the ability to customize AI solutions with locally accented speech data.

IMDA National Speech Corpus

The Infocomm Media Development Authority of Singapore has released a large dataset, which is:

• A 3 part speech corpus each with 1000 hours of recordings of phonetically-balanced scripts from ~1000 local English speakers.

• Audio recordings with words describing people, daily life, food, location, brands, commonly found in Singapore. These are recorded in quiet rooms using a combination of microphones and mobile phones to add acoustic variety.

• Text files which have transcripts. Of note are certain terms in Singlish such as ‘ar’, ‘lor’, etc.

This is a bounty for the open AI community in accelerating efforts towards speech adaptation. With such efforts, the trajectory for the local AI community and businesses are poised for major breakthroughs in Singlish in the coming years.

We have leveraged the IMDA national speech corpus as a starting ground to see how adding customized audio snippets from locally accented speakers drives up accuracy of transcription. An overview of the uptick is in the below chart. Without any customization, the holdout set performed with an accuracy of 73%. As more data snippets were added, we can validate that with the right datasets, we can drive accuracy up using human annotated speech snippets.

On the left is the uplift in terms of accuracy. The right correspondingly shows the Word Error Rate dropping on addition of more audio snippets

Keeping human in the loop

The speech recognition models learn from humans, based on “human-in-the-loop learning”. Human-in-the-Loop Machine Learning is when humans and Machine Learning processes interact to solve one or more of the following:

Making Machine Learning more accurate

Getting Machine Learning to the desired accuracy faster

Making humans more accurate

Making humans more efficient

An illustration of what a human in the loop looks like is as follows.

In a nutshell, human in the loop learning is giving AI the right calibration at appropriate junctures. An AI model starts learning for a task, which eventually can plateau over time. Timely interventions by a human in this loop can give the model the right nudge. “Transfer learning will be the next driver of ML success.”- Andrew Ng, in his Neural Information Processing Systems (NIPS) 2016 tutorial

Not everybody has access to volumes of call center logs, and conversation recordings collected from a majority of local speakers which are key sources of data to train localized speech transcription AI. In the absence of significant amounts of local accented data with ground truth annotations, and our belief behind transfer learning to be a powerful driver in accelerating AI development, we leverage existing models and maximize their ability to understand towards local accents.

The framework allows extensive room for human in the loop learning and can connect with AI models from both cloud providers and open source projects. A detailed treatment of the components in the framework include:

The speech to text model can be any kind of Automatic Speech Recognition (ASR) engine or Custom Speech API, which can run on cloud or on premise. The platform is designed to be agnostic to the ASR technology being used.

Search for ground truth snippets. In a lot of cases when the result is available, a quick search of the training records can point to the number of records trained, etc.

Breakdown on Word Error Rates (WER): The industry standard to measure Automatic Speech Recognition (ASR) systems is based on the Word Error Rate, defined as the below

where S refers to the number of words substituted, D refers to the number of words deleted, and I refer to the number of words inserted by the ASR engine.

A simple example illustrating this is as below, where there is 1 deletion, 1 insertion, and 1 substitution in a total of 5 words in the human labelled transcript.

Word Error Rate comparison between ground truth and transcript (Source: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-custom-speech-evaluate-data)

So, the WER of this result will be 3/5, which is 0.6. Most ASR engines will return the overall WER numbers, and some might return the split between the insertions, deletions and substitutions.

However, in our work (platform), we can provide a detailed split between the insertions, substitutions and deletions.

The platform built has ready interfaces that allow human annotators to plug audio files with relevant labeled transcriptions, to augment data

It ships with dashboards which show detailed substitutions, such as how often was the term ‘kaypoh’ transcribed as ‘people’.

The crux of the platform is the ability to control the existing transcription accuracy, by getting a detailed overview of how often the engine is having trouble transcribing certain vocabulary, and allowing human to give the right nudges to the model.

References and useful links