From classic Command Prompt to fully customizable Terminal

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Do you remember the Command Prompt? Are you still using it?

Command Prompt

There were so many customization options in there like font size, font type, and colors (all eight of them)!

Command Prompt Options

If like me, you need a bit more customization, I do have something for you.

Windows Terminal is the re-imagination of what a first-class command prompt experience should and gosh does it deliver. Let me show you what I got when I first launched the Command Prompt from the terminal.

Windows Terminal First Look around

Introducing the Amazing Windows Terminal

So the first thing I noticed? Transparency. I know it may sound superficial, but it’s catchy. The second thing? The tab with two buttons beside it. + will allow me to have more Command Prompt in here without changing windows(yay!).

The down arrow had me wondering for a second, so I clicked it.

So many options…

Your options may vary, but I have Windows Linux Subsystem installed on my machine and a few other options, so Ubuntu shows up. I’m amazed that I can use any of those prompts from a single option. Mind blown!

Back to Command Prompt

We can open up any shell/prompt/command line from Windows Terminal, which is nice, but how does it make Command Prompt better?

See that image above with the Setting option in there? Let’s click on that, and it opens up this file in Visual Studio Code.

Making the old feel new again

Settings.json of Windows Terminal

That’s a ton of JSON, but it will become quite easy quite fast.

The first link at Line 3 will bring you to the Official Docs, which is perfect. It will show you how to go in detail on every point.

I want to make this even more straightforward for you.

Do you see Line 6? Visual Studio Code will read the schema definition and enable code completion within your JSON file straight away.

If I wanted to modify something, I would create a new line and press “, and suddenly, you have all the options available.

Let me give you a new cmd.exe profile that you can overwrite and have something feels brand new right now.

{

    // Make changes here to the cmd.exe profile

    “guid”: “{0caa0dad-35be-5f56-a8ff-afceeeaa6101}”,

    “name”: “cmd”,

    “commandline”: “cmd.exe”,

    “useAcrylic”: true,

    “acrylicOpacity”: 0.7,

    “backgroundImageOpacity”: 0.7,

    “backgroundImageStretchMode”: “fill”,

    “backgroundImage”: “https://wallpapercave.com/wp/wp2053618.jpg”,

    “startingDirectory”: “C:git_ws”,

    “fontFace”: “Cascadia Code”,

    “fontSize”: 12,

    “hidden”: false

}

End Result

Refreshed Command Prompt

That doesn’t even closely look like our classic Command Prompt.

There are be many more options to cover that we can cover in another article. What we did for Command Prompt, we could do for every terminal/shell in our previous list.

Next Steps

Want to have a terminal that stands out? Do you want a terminal that is suited just for you and no one else?

• Follow our installation instructions, and you will be started in less than 5 minutes.


• After installing the Windows Terminal, start by copy/pasting settings from profiles you like in our Custom Terminal Gallery.


• Take a look at this Microsoft Learn Module on Windows Subsystem for Linux


• Take a look at this Microsoft Learn Module on Powershell

With those basics mastered, you are ready to tweak Windows Terminal until you feel at home with any shell.

If you create something unique, please share it with me on Twitter! I can’t wait to see your creations!

Satin: Microsoft’s latest AI-powered Audio codec for real-time communications

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Jigar Dani, Principal PM Manager, Microsoft
Sriram Srinivasan, Principal Software Engineering Manager, Microsoft

Over a decade ago Skype invented the Silk audio codec to transmit speech over the internet and catalyzed the voice over internet protocol (VoIP) industry. The primary codec used in VoIP then was G.722 that required 64 kbps to transmit wide band (16 kHz) speech, Silk on the other hand offered wideband quality starting at 14 kbps. Additionally, Silk was an adaptive variable bitrate codec that seamlessly switched from delivering narrow band (8 kHz) speech at ultra-low bandwidth of 6 kbps to offer a near transparent quality speech at higher bit rates. This was critical for dial-up and limited broadband internet that existed at that time and has served us well as the default codec for Skype and Microsoft Teams. It is also the basis of the voice mode of the OPUS codec which has been predominantly used in VoIP solutions in the last decade.

As we enter a new decade users have options to choose from several high-end connectivity alternatives such as high-speed broadband, optical fiber and 5G. Yet large segments of our user base are still limited to low cable internet speeds or 3/4G cellular networks. They encounter constrained network situations with over 50% packet loss and sporadic loss of coverage when moving between cell towers on commute or switching between network types. Network availability becomes unpredictable even when sharing internet at home with family members to stream video, gaming, work remotely and attend online schooling. Meanwhile, user expectations and essential need especially in the pandemic sometimes outpace the improvements in network connectivity. We have a need to communicate and collaborate on the go – on every device, every network, and every environment. Thus, efficient utilization of available bitrate is every bit as important today as it was in the dial-up world. Bitrate savings can be used to provide additional resiliency and/or improve experiences on other workloads like video and content sharing. We have considered these aspects to holistically address the challenges and deliver a virtual voice experience that is as good as talking in person even in ultra-low bandwidth and highly constrained network conditions.

Today we share details on our new AI powered audio codec – Satin, that can deliver super wide band speech starting at a bitrate of 6 kbps, and full-band stereo music starting at a bitrate of 17 kbps, with progressively higher quality at higher bitrates. Satin has been designed to provide great audio quality even under high packet loss. Here is the net effect of our improved resiliency algorithms and new Satin codec (use your favorite headset to hear the audio files):

We have built this codec with multiple decades of algorithmic experience combined with advanced machine learning techniques and in this blog we provide a deeper look at getting this codec ready for our users.

What’s narrowband, wideband, and super wideband voice?
Our ear can generally perceive sounds that range in frequency from 20 Hz to 20 kHz. When dealing with discrete time signals, we need to sample the audio waveform at a minimum of twice the highest frequency we wish to reproduce. This is generally why CD-quality music is sampled at 44.1 kHz (44100 samples per second) or 48 kHz. Early telephony systems used a sampling rate of 8 kHz and could reproduce frequencies up to 4 kHz (in practice up to 3.4 kHz), which was considered sufficient at the time for speech communication. While a lower sampling rate implies fewer bits per second to transmit over the wire, it resulted in the all too familiar tinny voice quality over the phone as the higher vocal frequencies present in natural speech could not be reproduced. VoIP solutions, which were no longer limited by the narrowband telephony infrastructure, introduced us to the magic of wideband speech (reproduce up to 8 kHz, sampled at 16 kHz) and users were immediately able to appreciate the crisper, more natural and intelligible sound.

Codecs such as Silk and Opus (the default audio codec in WebRTC) took this a step further with the introduction of super wideband voice, capturing frequencies up to 12 kHz, sampled at 24 kHz (energy drops off rapidly at frequencies above 12 kHz for human voice). As mentioned earlier, higher sampling rates imply a higher bitrate. Satin re-defines super wideband to cover frequencies up to 16 kHz (sampled at 32 kHz) for greater clarity and sibilance, and its efficient compression enables super wideband voice at 6 kbps.

Frequency components of the sound /t/ in the word “suit.” There is a significant amount of energy well beyond the narrowband cut-off of 4kHz and even the wideband cutoff of 8 kHz. Preserving energy in the higher spectral components results in more natural sounding speech.

Listen to the two samples below in your favorite headphones. The Satin super wideband speech sample sounds a lot more natural and intelligible, much like what you will hear when you are talking to someone in person.

Silk narrowband at 6 kbps: Satin super wideband at 6 kbps:

How do you get super wideband at 6 kbps?
To achieve super wideband quality at 6 kbps, Satin uses a deep understanding of speech production, modelling and psychoacoustics to extract and encode a sparse representation of the signal. To further reduce the required bitrate, Satin only encodes and transmits certain parameters in the lower frequency bands. At the decoder, Satin uses deep neural networks to estimate the high band parameters from the received low band parameters, and a minimal amount of side information sent over the wire. This approach solved the primary challenge of reproducing super wideband voice at ultra-low bitrates but introduced a new challenge of computational complexity. The analysis of the input speech signal to extract a low dimensional representation is computationally intensive. Real-time inference on deep neural networks adds to the complexity. The team then focused on reducing the complexity through both algorithmic optimizations as well as techniques such as loop vectorization beyond what the compiler could achieve. This resulted in close to a 40% reduction in computational complexity and allowed us to run on all our users’ devices.

As with all features, we A/B tested Satin before widely rolling it out – both to ensure there were no regressions, as well as to quantify the positive impact for our users. The A/B tests showed a high statistical significant increase in call duration for Satin compared to Silk at these low bitrates. Offline crowdsourced subjective tests to evaluate codec quality at 6 kbps showed the mean opinion score (MOS) rating of Satin to be 1.7 MOS higher than Silk.

How resilient is Satin to packet loss?
Yes, majority of our calls are on Wi-Fi and mobile networks, where packet loss is common and can adversely affect call quality. Satin is uniquely positioned to compensate for packet loss. Unlike most other voice codecs, Satin encodes each packet independently, so the effect of losing one packet does not affect the quality of subsequent packets. The codec is also designed to facilitate high quality packet loss concealment in an internal parametric domain. These features help Satin gracefully handle random losses where one or two packets are lost at a time.

Another type of packet loss, which is even more detrimental to perceived quality, is where several packets are lost in a burst. Here, Satin’s ability to deliver great audio at a low rate of 6 kbps provides the flexibility to use some of the available bitrate for adding redundancy and forward error correction that helps us recover from burst packet loss. Satin allows us to do this without having to compromise overall audio quality.

Satin is already used for all Teams and Skype two-party calls. We are rolling it out for meetings soon. Satin currently operates in wideband voice mode within a bitrate range of 6 – 36 kbps and will soon be extended to support full-band stereo music at a maximum sampling rate of 48 kHz. We are very excited for you to try this new codec, let us know what you think.

Subscribe to the Teams Engineering Tag RSS feed to stay in touch with the latest updates from our engineering teams.

Want to work on the team that builds bleeding edge AI technology: AI Jobs in M365 Intelligent Conversations and Communications Cloud Team

Container Image builds on Kubernetes clusters with Containerd and Azure DevOps self-hosted agents

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

Containerd is the default container runtime with AKS clusters on Kubernetes version 1.19 onwards. With a containerd-based node and node pools, instead of talking to the dockershim, the kubelet will talk directly to containerd via the CRI (container runtime interface) plugin, removing extra hops on the flow when compared to the Docker CRI implementation. As such, you’ll see better pod startup latency and less resource (CPU and memory) usage.

This change restricts containers from accessing the docker engine, /var/run/docker.sock, or use Docker-in-Docker (DinD).

In order to build docker images, Docker-in-Docker is a common technique used with Azure DevOps pipelines running in Self-Hosted agents. With Containerd, the pipelines building docker images no longer work and we need to consider other techniques. This article outlines the steps to modify the pipelines to perform image builds on Containerd enabled Kubernetes clusters.

Azure VM scale set agents is an option to scale self-hosted agents outside Kubernetes. To continue running the agents on Kubernetes, we will look at 2 options. One to perform image builds outside the cluster using ACR Tasks and another using kaniko executor image which is responsible for building an image from a Dockerfile and pushing it to a registry.

Building images using ACR Tasks

ACR Tasks facilitates container image builds.

Modify the existing pipelines/create a new pipeline to add an Azure CLI Task running the below command.

az acr build --registry <<registryName>> --image <<imageName:tagName>> .

The command will:

• Run in the current workspace


• Package the code and upload to in a temp volume attached to ACR Tasks


• Build the container image


• Push the container image to the registry

The pipeline should look as illustrated below:

Though this approach is simple, it has a dependency on ACR. The next option deals with in-cluster builds which does not require ACR.

Building images using Kaniko

To use Kaniko to build images, it needs a build context and the executor instance to perform the build and push to the registry. Unlike Docker-in-Docker scenario, Kaniko builds are executed in a separate pod. We will use Azure Storage to exchange the context (source code to build) between the agent and the kaniko executor. Below are the steps in the pipeline.

• Package the build context as a tar file


• Upload the tar file to Azure Storage


• Create a pod deployment to execute the build


• Wait for the Pod completion to continue

The script to perform the build is as below:

# package the source code
tar -czvf /azp/agent/_work/$(Build.BuildId).tar.gz .

#Upload the tar file to Azure Storage
az storage blob upload --account-name codelesslab --account-key $SKEY --container-name kaniko --file /azp/agent/_work/$(Build.BuildId).tar.gz --name $(Build.BuildId).tar.gz

#Create a deployment yaml to create the Kaniko Pod
cat > deploy.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kaniko-$(Build.BuildId)
  namespace: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=Dockerfile"
    - "--context=https://<<storageAccountName>>.blob.core.windows.net/<<blobContainerName>>/$(Build.BuildId).tar.gz"
    - "--destination=<<registryName>>/<<imageName>>:k$(Build.BuildId)"
    volumeMounts:
    - name: docker-config
      mountPath: /kaniko/.docker/
    env:
    - name: AZURE_STORAGE_ACCESS_KEY
      value: $SKEY
  restartPolicy: Never
  volumes:
  - name: docker-config
    configMap:
      name: docker-config
EOF

The storage access key can be added as an encrypted pipeline variable. Since the encrypted variables are not passed on to the tasks directly, we need to map them to an environment variable.

As the build is executed outside the pipeline, it is required to monitor the status of the pod to decide on the next steps within the pipeline. Below is a sample bash script to monitor the pod:

# Monitor for Success or failure

while [[ $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') != "Succeeded" && $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') != "Failed" ]]; do echo "waiting for pod" && sleep 1; done

# Exit the script with error if build failed

if [ $(kubectl get pods kaniko-$(Build.BuildId) -n kaniko -o jsonpath='{..status.phase}') == "Failed" ]; then 
    exit 1;
fi

The complete pipeline should look similar to below:

Task 1: [Optional ] Get the KubeConfig (If not supplied through secrets)

Task 2:  [Optional ] Install Kubectl latest (if not installed with the agent image)

Task 3: Package Context and Prepare YAML

Note how the pipeline variable is mapped to the Task Environment variable

Task 4: Create the Executor Pod

Note: Alternatively, can be included in the script kubectl apply -f deploy.yaml

Task 5: Monitor for Status

Summary

These build techniques are secure compared to Docker-in-Docker scenario as no special permission, privileges or mounts are required to perform a container image build.

Blast Off With Azure Advocates: Presenting The Azure Space Mystery

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

TLDR; Azure Space Mystery is an interactive experience teaching you about Space, women scientist and how you can interact with LEARN to solve mysteries in the game. Blast off for the Azure Space Mystery

 

For Educators and Students:

This experience points to LEARN (aka.ms/learn) content on both Neural Networks, APIs and getting started with C# in an interactive environment (IDE not needed) 

-Neural Network https://docs.microsoft.com/en-us/learn/modules/train-test-predictive-ai-model-nasa/3-build-neural-network?WT.mc_id=academic-11769-cxa

– APIs https://docs.microsoft.com/en-us/learn/modules/use-apis-discover-museum-art/4-query-met-api?WT.mc_id=academic-11769-cxa

– C# interactive tutorial https://docs.microsoft.com/en-us/learn/modules/csharp-basic-operations/3-exercise-math-operators?WT.mc_id=academic-11769-cxa

 

 

 

 

Location: 400 km above Earth, traveling at 27,600 km/h. The space crew is in a good mood. They are looking forward to today’s return to Earth. We have made groundbreaking discoveries that will change the way we understand the …

 

Suddenly, a voice emanates from your communication console. “Captain, we have received an SOS message from the International Space Station. Their solar array wing was knocked off by debris. They are quickly running out of power! They need our help to collect the four missing pieces and deliver them back to the ISS as soon as possible.”

When that SOS call comes from the International Space Station, you know that you’re the person for the job!

Push the buttons and start your adventure! Will you embark on either the Rosetta, SOHO, Magnet, or Cluster Missions?

Azure Advocates and Community Advocacy PMs are excited to offer our third mystery experience, the Azure Space Mystery! Following on the Azure Mystery Mansion and the Azure Maya Mystery, this adventure sends you on missions in space to collect the four missing pieces of the wing. Store each piece in your space ship’s Collection Bay and find your way to the ISS to save the day.

 

During your mission, you will have to solve code challenges and unlock elements of the ship to collect the items. Can you figure out the circumference of the spool around which you must wrap the missing wire? Can you find the keyword on Microsoft Learn to unlock the door so you can complete your space walk? What if you fly into the tail of a comet?

Curious how we built this game? It uses the same architecture as the Azure Maya Mystery: TailwindCSS, VuePress, and an Azure Static Web App.

Every great explorer finds helpers along the way, and the Space Mystery is no different. You will be helped at strategic moments by four famous women who, in history, helped advance scientific inquiry. You’ll have to play the game to discover who they are, but get ready to meet a mathematician, a pilot, a scientist, and an astronomer whose work spanned 14 centuries.

You’ll not only meet these inspiring scientific women in the game, but you can also meet them in Minecraft!

 

The connection between the Space Mystery game and Minecraft is made by your acquisition of a final badge, available when you complete your missions. Collect your Space Learner badger badge and use it in the MyMetaverse Minecraft server.

In the server, this badger token will give you exclusive access to a Heroes Hangout dedicated for Azure Heroes users. The server is accessible in Minecraft Java edition at mc.mymetaverse.io. To use the badge in the server, link an Enjin Wallet, which is the app where Azure Heroes tokens are stored.

 

The Azure Space Mystery was brought to you in honor of the 6th International Day for Women and Girls in Science. This day was founded by the “Space Princess”: H.R.H. Princess Dr. Nisreen El-Hashemite. It is our hope that our game will teach a little about other famous and impactful women in Science.

But wait, there’s more! Download free wallpapers of our bespoke cosmic art for your video call backgrounds!

 

We would like to acknowledge the folks who contributed to the content of the game: Marc Duiker who did the pixel art, and Dr. Mark Looper, who provided space-focused technical expertise. Many thanks to the ‘mystery team’ of Cloud Advocates, Chris Noring and myself (Jen) and to our mysterious PM team, in particular Lucie Simeckova, Floor Drees, and Adam Jackson, Eva Amezua de Casado, Jan Schenk, Adi Stein Ben-Nun, and Cynthia Zanoni for overseeing production.

Are you ready to accept your mission to explore space? Let’s go! Blast off for the Azure Space Mystery

How to send messages to or receive from Service Bus/Event Hub with Service Bus Explorer?

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

Service Bus Explorer is an open-source tool, created with Microsoft supported .NET SDKs and available on any computer with the .NET framework.  Service Bus Explorer allows users to connect to a messaging namespace (Service Bus, Event Hub, Notification Hub, and Relay) and administer their messaging entities through a GUI. The tool provides advanced features like import/export functionality or the ability to test topic, queues, subscriptions, relay services, notification hubs and events hubs.

Download the latest version from link below.  It is standalone so no installation is required, just unzip and run the ServiceBusExplorer.exe executable from the folder.

https://github.com/paolosalvatori/ServiceBusExplorer/releases

Connect to a namespace

1. Go to the corresponding Service Bus or Event Hub namespace in the Azure Portal.  Click on Settings-> “Shared Access Policies” and click on RootManageSharedAccessKey.  Copy the Primary Connection String and keep this value handy.
   NOTE:  You need to use a connection string with Manage Claims with Service Bus Explorer 

2. Start Service Bus Explorer, go to “File” and click on “Connect” and in the popup window, choose “Enter connection string…” from Service Bus Namespace dropdown. And in the Connection Settings pane, under Connection String, enter Primary Connection String from step 1) and click on “OK.”

Service Bus – Sending messages to Queues or Topics

1. To send a message to a Queue or a Topic, right click on the queue or topic name on the Service Bus Explorer navigation pane and select “Send Messages”.

1. In the “Send Messages” popup dialog box, enter the message in the text box, select the message format and click “Start” to send the message:

** You can specify message properties in the “Senders” tab, and attach files to the message from “Files” tab.

Service Bus – Receiving messages from Queues or Topic Subscriptions

1.       To peak or receive a message from a Queue or a topic subscription, right click on the queue or subscription name on the Service Bus Explorer navigation pane and select “Receive Messages”.

2.       In the popup dialog box, it is possible to peek or receive a configurable number of messages from a queue or topic subscription:

3.       When you click the Ok button in the Retrieve messages from queue (or topic subscription) dialog, messages are retrieved and shown in the following tab:

Service Bus – Check the deadletter reason for specific messages

1.       Once you have successfully connected to your Service Bus namespace, select one of your queue as shown below.  You can also do this for a topic subscriptions.

2.       Just like receive active messages shown in the previous section, you can choose to peak or receive deadlettered messages in the popup dialog box.  Select “Peek” to peek some messages from the deadletter and click “Ok”

3.       You can then review the deadletter reason for each individual message as shown below
 

Event Hub – Sending Event messages to a Event Hub

1.       To send a event message to a event hub, right click on the event hub name on the Service Bus Explorer navigation pane and select “Send Events”.

2.       In the “Send Events” popup dialog box, enter the message in the text box, select the message format and click “Start” to send the message:

** You can specify message properties in the “Sender” tab, and attach files to the message from “Files” tab.

Event Hub – View messages through a consumer group

1. Once you have successfully connected to your Event Hub namespace, drill down to one of your consumer groups, right click on it, and select “Create Consumer Group Listener.”
   NOTE:  If your Event Hub is actively being read from, you will want to use a different consumer group from the one(s) actively being used
    

2. In the listener dialog popup, click “Start” to start the consumer group listener:

NOTE: You can specify the “Starting Date Time UTC” to selectively fetch event data enqueued after this point, and check “Verbose” option to enable verbose logs.

1. Check the Log at the bottom of the screen and you can see the message receiving logs. You can also select the Events tab and check for the message body to check the actual events that are in your Event Hub.
    

Lesson Learned #162: Cannot show requested dialog obtaining properties database in Managed Instance

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Today, I saw this error message when our customer is trying to obtain the properties of a database. This issue sometimes happened when SQL Server Management Studio is trying to obtain the last backup date done in Azure SQL Managed Instance. 

For example, using SQL Server Profiler or Extended Event Profiler I was able to find the TSQL that SQL Server Management Studio is running, obtaining the same error message. 

exec sp_executesql N'
        create table #tempbackup (database_name nvarchar(128), [type] char(1), backup_finish_date datetime)
        insert into #tempbackup select database_name, [type], max(backup_finish_date) from msdb..backupset where [type] = ''D'' or [type] = ''L'' or [type]=''I'' group by database_name, [type]
SELECT
(select backup_finish_date from #tempbackup where type = @_msparam_0 and db_id(database_name) = dtb.database_id) AS [LastBackupDate]
FROM
master.sys.databases AS dtb
WHERE
(dtb.name=@_msparam_1)
        drop table #tempbackup
',N'@_msparam_0 nvarchar(4000),@_msparam_1 nvarchar(4000)',@_msparam_0=N'D',@_msparam_1=N'database name'

In this situation, as the backup system that Azure SQL Managed Instance is different than other ones, in order to fix this issue, basically, you need to run the following command: 

msdb..sp_delete_database_backuphistory '<database_name>'

After it, you are going to be able to obtain the properties of the database. 

Enjoy!

Unable to access SQL Database connected via Service Endpoint after failover occurs

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Some customer asked me about the following query around their creating system.

Query

“We are creating some system and using paired regions to secure redundancy of it. Each database instance (primary and replicated) is located on both regions and these instances belongs to a fail over group. When we tested database failover, applications in their environment could not access primary database instance. We don’t think network is reachable since global peering is configured between virtual networks in each region. What is the root cause of this issue? How do we fix this issue?”

Backgrounds

As backgrounds are not clear, I asked the customer to share details about their system and the facing issue. 

• Their system is deployed to paired regions to secure the redundancy of their system.


• Traffic Manager works in front of their system to load balance incoming traffic. They use priority-based traffic-routing for load balancing. If some failure occurs in active region, Traffic Manager changes route of incoming traffic to the another region. 


• Global peering between virtual networks in both regions is configured.


• They use App Service to host their applications. Their App Service instances are integrated with virtual networks, and service endpoints for SQL Database instances are configured at the subnets where these app services are integrated. Also, service endpoints for App Service instances are configured in order to interact each App Service instance.


• They use SQL Database in this system and instances on both regions belongs to automatic failover group. As read-write/read-only listener is geo-independent, they don’t have to modify database connection string used in applications whenever database failover occurs.


• As of now, they don’t mind that primary database region should be the same as the one where Traffic Manager routes incoming traffic. In other words, they think cross region connection is fine.

The following diagram reflects their comments and our hearing results.

Root cause

If you are familiar with Azure, you can detect the root cause of this issue easily. This is due to service endpoint limitation. For Azure SQL, a service endpoint applies only to Azure service traffic within a virtual network’s region.

The following case works fine.

However, the following case does not work even if global peering is configured.

Solutions

In this case, we can choose two options listed below.

• Using private link


• Modifying traffic routing rule

1. Using private link

If cross region connection is still fine, they can fix this issue by using private link instead of service endpoint.

Azure Private Link for Azure SQL Database and Azure Synapse Analytics

https://docs.microsoft.com/azure/azure-sql/database/private-endpoint-overview

When using private link, the diagram looks like this.

When using private link, the following limitations should be considered.

Cost

• Private link is not free of charge.
  https://azure.microsoft.com/pricing/details/private-link/


• Costs for network traffic across regions are required.

Performance

• When connecting SQL Database, all connections are proxied via the Azure SQL Database gateways. That leads to poorer throughput than direct connection.
  https://docs.microsoft.com/azure/azure-sql/database/connectivity-architecture#connection-policy 


• Network latency gets longer in case of cross region connection.

2. Modifying traffic routing rule

In some cases, private link does not meet requirements. In this case, we should configure Traffic Manger to match between the region where Traffic Manager routes incoming traffic and database primary region. The diagram looks like this.

To achieve this, the following configuration is required.

• First of all, priority for active region is set smaller value (e.g. 50) , and the priority for the other region is set much bigger value (e.g. 1000).number. This configuration allows incoming traffic to be routed to active region. For more details, see the following document.

Priority traffic-routing method
https://docs.microsoft.com/azure/traffic-manager/traffic-manager-routing-methods#priority-traffic-routing-method

• Then, healthcheck API should be configured. The API checks if access between applications and databases is healthy. If heathy, the API returns HTTP 200, otherwise, it returns 503.


• Following the document, traffic Manager is configured in order to use this API to monitor endpoint. If healthcheck API returns 503, Traffic Manger modifies routing route.

Configure endpoint monitoring

https://docs.microsoft.com/azure/traffic-manager/traffic-manager-monitoring#configure-endpoint-monitoring

This concept has some limitations listed below.

• Needless to say, healthcheck API should be created. 


• It takes some time to change routing region. Precisely, the minimum number of trials (from 0 to 9) to monitor endpoint by healthcheck API and trial interval (default is 30 second interval, and 10 second interval is also available, but additional cost is required). For more details, see the following document.

Configure endpoint monitoring
https://docs.microsoft.com/azure/traffic-manager/traffic-manager-monitoring#configure-endpoint-monitoring

Conclusion

In this case, I suggested both ways and asked this customer to make their decision. And last but not least, Traffic manager is used in this case, but this solution is applicable when using Azure Front Door.

Optimize your Azure Costs by Automating SAP System Start – Stop

by Contributed | Feb 15, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

We are happy to share with the SAP on Azure community one solution to automate your SAP system start / stop in Azure.

This is ready to use, flexible, end-to-end solution (including PaaS Azure automation runtime environment, scripts, and runbooks, tagging process etc.) that enables you to automatically:

• Start / Stop your SAP systems, DBMSs, and VMs.


• SAP application servers


• If you use managed disks (Premium and Standard), you can decide to convert them to Standard during the stop procedure, and to Premium during the start procedure.

This way you achieve cost savings both on the compute and on the storage side!

SAP systems stop and SAP application servers stop is specially designed for a graceful shutdown, allowing SAP users and batch jobs to finish. This way you can minimize the SAP system or the SAP application server’s downtime impact.  The approach is similar on the DBMS side.

To further enhance the user experience, you can consume this functionality using a cool SAP Azure Power App . For more information you can check a great blog of Martin Pankraz –   Hey, SAP Systems! My PowerApp says Freeze! But only if you’re done yet .

This solution is the product of a joint effort of the SAP on Azure CAT Team (Cloud Advisory Team) andthe SAP on Azure FastTrack Team (Robert Biro  and Martin Pankraz).

Solution Capabilities

In details, with this solution you can do the following:

• Start / Stop a complete SAP NetWeaver Central System on Linux or Windows, and the VM.
  The typical scenario here is a non prod SAP systems.
  
  


• Start / Stop of a complete SAP NetWeaver Distributed System on Linux or Windows and VMs.
  The typical scenario here is a non prod SAP systems.
  
  Here you have:
  
  
  
  • One DBMS VM (HA is currently not implemented)
  
  
  • One ASCS/SCS or DVEBMGS VM (HA is currently not implemented)
  
  
  • One or more SAP application servers
  
  
  • It is assumed that SAPMNT file share is located on SAP ASCS or DVEBMGS VM.
    
    
  
  
  
  


• In a distributed SAP landscape, you can deploy your SAP instances across multiple VMs, and those VMs can be placed in different Azure resources groups.
  
  


• In a distributed SAP landscape, the SAP application instances (application server and SAP ASCS/SCS instance) can be on Windows and DBMS on Linux (this is so called heterogenous landscape), for DBMS that support such scenario for example SAP HANA, Oracle, IBM DB2, SAP Sybase and SAP MaxDB.
  
  


• On the DBMS side, starting, stopping, getting the status of DBMS itself is implemented for:
  
  
  
  • SAP HANA DB
  
  
  • Microsoft SQL Server DB
    
    
  
  
  
  


• Currently, starting, stopping and getting the status of DBMS is NOT implemented for Oracle, IBM DB2, Sybase and MaxDB DBMSs.
  
  You can use the solution with these DBMSs, but you  need to make sure that:
  
  
  
  • DBMS is configured to automatically start with OS start.
  
  
  • SAP system in startup procedure start first DBMS (which is default SAP start order in SAP instance profile)
  
  
  
  

Although we did not test with all these DBMSs, the expectation is that this approach will work.

• Start / Stop Standalone HANA system and VM
  
  


• Start / Stop SAP Application Server(s) and VM(s)
  This functionality can be used for SAP application servers scale out -scale in process.   
  
  One meaningful scenario for production SAP systems is that you, as an SAP system administrator,  identify upcomming peaks in the system load (for example Black Friday or Year-End close), where you know in advance how many SAPS / application servers you would need to meet the additional load requirements, and for how long. Then you can either schedule start / stop or manually start/stop a number of already prepared SAP application servers, that will cover the load peak.
  
  


• Converting the disks from Premium to Standard managed disks during the stop process, and the other way around during the start process to reduce the storage costs.
  
  


• Start / Stop actions can be executed manually, or can be scheduled.
  
  
  
  
  
   
  
  

PaaS Solution Architecture

The solution is using Azure automation account PaaS as an automation platform to execute the SAP shutdown/startup jobs.

Runbooks are written in PowerShell. A PowerShell module SAPAzurePowerShellModules is used by all runbooks. These runbooks and the module are stored in PowerShell Gallery, and are easy to import.

Information about the SAP landscape and instances are stored in VM Tags. VM tagging can be done either manually or even better – using prepared tag runbooks.

<sid>adm password is needed on Windows OS and is stored securely in the Credentials area of the Azure automation account.

Secure assets in Azure Automation include credentials, certificates, connections, and encrypted variables. These assets are encrypted and stored in Azure Automation using a unique key that is generated for each Automation account. Azure Automation stores the key in the system-managed Key Vault.

The Starting and Stopping of:

• An SAP system and an SAP Application server is implemented using scripts (calling SAP sapcontrol executable).


• An SAP HANA DB is implemented using scripts (calling SAP sapcontrol executable).


• SQL Server DB start / stop / monitoring is implemented using scripts (calling SAP Host Agent executable).


• All scripts are executed at OS level,  in a secure way via Azure VM agent.


• All SAP system and DBMS start / stop / monitor scrips are generated on the fly during the runtime, therefore there is no need to store them anywhere.

Soft / Graceful Shutdown of SAP System, Application Servers, and DBMS

SAP System and SAP Application Servers

The stopping of the SAP system or SAP application server will be done using an SAP soft shutdown or graceful shutdown procedure, within a specified timeout. The SAP soft shutdown is handling gracefully SAP processes, users, etc. within the specified downtime time, during the stop of the whole SAP system or one SAP application server.

Users will get a popup to log off, SAP application server(s) will be removed from different logon groups (users, batch,  RFC,  etc.), the procedure will wait for SAP batch jobs to complete (until the specified timeout is reached). This functionality is implemented in the SAP kernel.

INFO: You can specify the value for SAP soft shutdown time as a parameter. The default value is 300.

DBMS Shutdown

For SAP HANA and SQL Server, DB soft shutdown is also implemented, which will gracefully stop these DBMS, so DB will have time to flush consistently all content from memory to storage and stop all DB processes.

User Interface Possibilities

A user can trigger start / stop in two ways:

• using Azure Automation account portal UI
  
  
  
   
  
  

  

  
  
  
  


• or via modern SAP Azure Power App, which can be consumed in a browser,  smart phones or Teams:
  
  
   
  
  

  

  
  
  
  For more information you can check a great blog of Martin Pankraz – Hey, SAP Systems! My PowerApp says Freeze! But only if you’re done yet.
  
  The SAP PowerApp application is fully integrated with backend start / stop functionality with Azure automation account. It will automatically collect information on all available SAP SIDs (via SAPSID tag) and offer Start / Stop / SAP system status functionality!
  
  

Cost Optimization Potential

Cost Savings on Compute for non-Productive SAP Systems

The non-production SAP systems, like dev, test, demo, training etc., typically do not run 24/7. Lets assume you would run them 8 hours per day, Monday through Friday. This means you run and pay for each VM 8 hours x 5 days = 40 hours. The rest of the time of 128 hours per week you don’t need to pay, which translates into approximatelly 76 % of savings on compute!

Cost Savings on Compute for Productive SAP Systems

Productive SAP system run typically 24/7 and you never completely shut them down.  Still, there is a huge potential for savings in the SAP application layer. The SAP application layer constitutes the largest portion of SAPS in an SAP system.

INFO: SAP Application Performance Standard (SAPS) is a hardware-independent unit of measurement that describes the performance of a system configuration in the SAP environment. It is derived from the Sales and Distribution (SD) benchmark, where 100 SAPS is defined as 2,000 fully business processed order line items per hour. SAPS is an SAP measure of compute process power.

In an on-premises environment, an SAP system is often oversized,  so that it can process the required peak loads. But the reality is, these peaks are rare (maybe few days in 3 months). Most of the time such systems are underutilized. I’ve seen prod systems that have 5 – 10 % of total CPU utilization most of the time.

In the cloud we have the possibility to run only what we need, and pay for what we used – hence, the SAP application servers’ layer is a perfect candidate to bring down the cost for SAP productive systems!

Here, this solution offers jobs to start / stop an SAP application server and VMs. It is using a soft shut down, allowing SAP users and processes enough time to complete.

Cost Savings on Storage

If you use Premium storage, there is an opportunity for cost savings, by converting such managed storage to standard, while the system is not running.

Let’s say you need to use Premium storage (especially for the DBMS layer) to get a good SAP performance during the runtime. But once the SAP system (and VMs) is stopped, if you choose to convert the disks from Premium to Standard disks, you will pay much less on the storage during the time the system and the VMs are stopped.

During the start procedure, you can decide to convert the disks back to premium, to have good performance while the SAP systems are running , and only pay for the more expensive Premium storage, while the system is running.

For example, if the SAP system would run 8 hours x 5 days = 40 hours, and the SAP system is stopped for 128 hours per week, that means that 128 hours per week, you will not pay the price for Premium storage, but the reduced price for Standard storage.

For example, price of 1 TB P30 disk approximately 2 times higher than 1 TB S30 Standard HDD disk. For above mentioned scenario, savings on 1 TB managed disk would be approximately 54 %!

For the exact updated pricing for managed disks, check here.

Cost of Azure Automation Account PaaS Service

The cost of using Azure automaton service is extremely low.   Billing for jobs is based on the number of job run time minutes, used in the month. And for watchers, billing is based on the number of hours used in a month. Charges for process automation are incurred whenever a job or watcher runs. You will be billed only for minutes/hours that exceed the free included units (500 min for free). If you exceed the monthly free limit of 500 min, you will pay per minute €0.002/minute.

Let’s say one SAP system start or stop takes on average 15 minutes. And let’s assume you scheduled your SAP system to start every morning from Monday to Friday and stop in the evening as well. That will be 10 executions per week, and 40 per month for one SAP system.

This means that in 500 free minutes you can execute 33 start or stop procedures for free.

Everything extra you need to pay. For one start or stop (of 15 min), you would pay 15 min * €0.002 = €0.03. And for 40 start / stop of ONE SAP system you would pay € 1.2 per month!

For uptodate pricing, you can check here.

Solution Cost of Azure Automation Account Management

Often, when you use a solution which offers you specific functionality, you need to manage it as well, learn it, etc. All this generates additional costs, which can be quite high. 

As Azure automation account is a PaaS service, you have here ZERO management costs!

Plus, it is easy to set it up and use it.

Documentation and Scrips

All documentation and scripts can be found here on GitHub. All job scripts and SAP on Azure PowerShell Module is available on PowerShell Gallery.