Making MsQuic Blazing Fast

by Contributed | Apr 14, 2021 | Technology

This article is contributed. See the original author and article here.

It’s been a year since we open sourced MsQuic and a lot has happened since then, both in the industry (QUIC v1 in the final stages) and in MsQuic. As far as MsQuic goes, we’ve been hard at work adding new features, improving stability and more; but improving performance has been one of our primary ongoing efforts. MsQuic recently passed the 1000^th commit mark, with nearly 200 of those for PRs related to performance work. We’ve improved single connection upload speeds from 1.67 Gbps in July 2020 to as high as 7.99 Gbps with the latest builds*.

* Windows Preview OS builds; User-mode using Schannel; and server-class hardware with USO.
** x-axis above reflects the number of Git commits back from HEAD.

Defining Performance

“Performance” means a lot of different things to different people. When we talk with Windows file sharing (SMB), it’s always about single connection, bulk throughput. How many gigabits per second can you upload or download? With HTTP, more often it’s about the maximum number of requests per second (RPS) a server can handle, or the per-request latency values. How many microseconds of latency do you add to a request? For a general purpose QUIC solution, all of these are important to us. But even these different scenarios can have ambiguity in their definition. That’s why we’re working to standardize the process by which we measure the various performance scenarios. Not only does this provide a very clear message of what exactly is being measured and how, but it has also allowed for us to do cross-implementation performance testing. Four other implementations (that we know of) have implemented the “perf” protocol we’ve defined.

Performance-First Design

As already mentioned above, performance has been a primary focus of our efforts. Since the very start of our work on QUIC, we’ve had both HTTP and SMB scenarios driving pretty much every design decision we’ve made. It comes down to the following: The design must be both performant for a single operation and highly parallelizable for many. For SMB, a few connections must be able to achieve extremely high throughput. On the other hand, HTTP needs to support tens of thousands of parallel connections/requests with very low latency.

This design initially led to significant improvements at the UDP layer. We added support for UDP send segmentation and receive coalescing. Together, these interfaces allow a user mode app to batch UDP payloads into large contiguous buffers that only need to traverse the networking stack once per batch, opposed to once per datagram. This greatly increased bulk throughput of UDP datagrams for user mode.

These design requirements have led to some significant complexity internal to MsQuic as well. The QUIC protocol and the UDP (and below) work are separated onto their own threads. In scenarios with a small number of connections, these threads generally spread to separate processors, allowing for higher throughput. In scenarios with a large number of connections, effectively saturating all the processors with work, we do additional work improves parallelization.

Those are just a few of the (bigger) impacts our performance-driven design has had on MsQuic architecture. This design process has affected every part of MsQuic from the API down to the platform abstraction layer.

Making Performance Testing Integral to CI

Claiming a performant design means nothing without data to back it up. Additionally, we found that occasional, mostly manual, performance testing led to even more issues. First off, to be able to make reasonable comparisons of performance results, we needed to reduce the number of factors that might affect the results. We found that having a manual process added a lot of variability to the results because of the significant setup and tool complexity. Removing the “middleman” was super important, but frequent testing has been even more important. If we only tested once a month, it was next to impossible to identify the cause of any regressions found in the latest results; let alone prevent them from happening in the first place. That inevitably led to a significant amount of wasted time trying to track down the problem. All the while, we had regressed performance for anyone using the code in the meantime.

For these reasons, we’ve invested significant resources into making performance testing a first-class citizen in our CI automation. We run the full performance suite of tests for every single PR, every commit to main, and for every release branch. If a pull request affects performance, we know before it’s even merged into main. If it regresses performance, it’s not merged. With this system in place, we have pretty much guaranteed performance in main will only go up. This has also allowed us to confidently take external contributions to the code without fear of any regressions.

Another significant part of this automation is generating our Performance Dashboard. Every run of our CI pipeline for commits to main generates a new data point and automatically updates the data on the dashboard. The main page is designed to give a quick look at the current state of the system and any recent changes. There are various other pages that can be used to drill down into the data.

Progress So Far

As indicated in the chart at the beginning, we’ve had lots of improvements in performance over the last year. One nice feature of the dashboard is the ability to click on a data point and get linked directly to the relevant Git commit used. This allows us to easily find what code change caused the impacted performance. Below is a list of just a few of the recent commits that had the biggest impact on single connection upload performance.

• d985d44 – Improves the flow control tuning logic


• 1f4bfd7 – Refactors the perf tool


• ec6a3c0 – Fix a kernel issue related to starving NIC packet buffers


• be57c4a – Refactors how we use RSS processor to schedule work


• 084d034 – Refactors OpenSSL crypto abstraction layer


• 9f10e02 – Switches to OpenSSL 1.1.1 branch instead of 3.0


• ee9fc96 – Adds GSO support to Linux data path abstraction


• a5e67c3 – Refactors UDP send logic to run on data path thread

Most of these changes came about from this simple process:

1. Collect performance traces.


2. Analyze traces for bottlenecks.


3. Improve biggest bottleneck.


4. Test for regressions.


5. Repeat.

This is a constantly ongoing process to always improve performance. We’ve done considerable work to make parts of this process easier. For instance, we’ve created our own WPA plugin for analyzing MsQuic performance traces. We also continue to spend time stabilizing our existing performance so that we can better catch possible regressions going forward.

Future Work

We’ve done a lot of work so far and come a long way, but the push for improved performance is never ending. There’s always another bottleneck to improve/eliminate. There’s always a little better/faster way of doing things. There’s always more tooling that can be created to improve the overall process. We will continue to put effort into all these.

Going forward, we want to investigate additional hardware offloads and software optimization techniques. We want to build upon the work going on in the ecosystem and help to standardize these optimizations and integrate it them into the OS platform and then into MsQuic. Our hope is that we will make MsQuic the first choice for customer workloads by bringing the network performance benefits QUIC promises without having to make a trade-off with computational efficiency.

As always, for more info on MsQuic continue reading on GitHub.

Royal Bank of Canada using Always Encrypted with secure enclaves

by Contributed | Apr 14, 2021 | Technology

This article is contributed. See the original author and article here.

Earlier this year, we announced the preview of Always Encrypted with secure enclaves in Azure SQL Database – the feature designed to safeguard sensitive data from malware and unauthorized users by enabling rich confidential queries.

Royal Bank of Canada (RBC) is one of the customers who are already using Always Encrypted with secure enclaves. For details, see Microsoft Customer Story – RBC creates relevant personalized offers while protecting data privacy with Azure confidential computing.

For more information about Always Encrypted with secure enclaves in Azure SQL Database, see:

• Inside Azure Datacenter Architecture with Mark Russinovich (the segment on Always Encrypted)


• A webinar including a deep dive on Always Encrypted with secure enclaves


• Tutorial: Getting started with Always Encrypted with secure enclaves in Azure SQL Database


• Always Encrypted with secure enclaves – documentation

Deploying M365 Learning Pathways in your GCC High Tenant

by Contributed | Apr 14, 2021 | Technology

This article is contributed. See the original author and article here.

Contributors:

Rob Garrett – Sr. Customer Engineer, Microsoft Federal 

John Unterseher – Sr. Customer Engineer, Microsoft Federal

Martin Ballard – Sr. Customer Engineer, Microsoft Federal 

This article replaces the previous article, which used the – now legacy – version of PnP PowerShell.

What are Learning Pathways?

Microsoft 365 learning pathways is a customizable, on-demand learning solution designed to increase usage and adoption of Microsoft 365 services in your organization. Learning Pathways consists of a fully customizable SharePoint Online Communication site collection, with content populated from the Microsoft online catalog; so, your content is always up to date. Learning Pathways provide integrated playlists to meet the unique needs of your organization.

M365 Learning Pathways build atop of the Look Book Provisioning Service and templates (https://lookbook.microsoft.com).  In a previous blog post, we detailed the nuances of the Look Book Provisioning Service and additional steps required to deploy templates to GCC High tenants. Since Learning Pathways depend on the provisioning service to create a Communication site with customizations, via a Look Book template, this post details the additional steps to follow those from the earlier blog post.

Challenge – Using Learning Pathways in GCC High

Microsoft strives to implement functionality parity between all sovereign clouds. However, since each Office 365 cloud type serves a different customer audience and requirements, functionality will differ between these cloud types. Of the M365 clouds – Commercial, Government Community Cloud, Government Community Cloud High, and DOD Cloud, the last two offer the least functionality to observe US federal mandates and compliance.

As Microsoft develops new functionality for Microsoft 365 and Azure clouds, we typically release new functionality to commercial customers first, and then to the other GCC, GCC High, and DOD tenants later as we comply with FedRAMP and other US Government mandates. Open-source offerings add another layer of complexity since open-source code contains community contribution and is seldom developed with government clouds in mind.

Apply Learning Pathways to GCC High

Microsoft 365 Learning Pathways offers manual steps to support deployment to an existing SharePoint Online Communication site. Recall from the earlier blog post that the Look Book Provisioning Service is unable to establish a new site collection in GCC High, because of necessary restrictions. We, therefore, deploy Learning Pathways using the manual steps with a pre-provisioned Communication site collection.

Manual setup of Learning Pathways requires experience working with Windows PowerShell and the PnP PowerShell module.

Prerequisites

Before getting into the manual steps, we must meet prerequisites for manual install of Learning Pathways, the following is a summary:

• Create and designate a new Communication in SharePoint Online for Learning Pathways.


• Create a tenant-wide application catalog (steps below).


• Install the latest SharePoint PnP.


• Perform all steps as a SharePoint Tenant Administrator.

We begin by creating a new Communication site via the SharePoint Administration site:

https://mytenant-admin.sharepoint.us/_layouts/15/online/AdminHome.aspx#/siteManagement/view/ALL%20SITES

Ensure the appropriate permissions for users of the Learning Pathways site:

4. Open the Learning Pathways site collection in your web browser.


5. From the home page, click the Share link.


6. Add students to the Site Visitors group.


7. Add playlist editors of the pathways site to the Site Members group.


8. Add site administrators of the pathways site to the Site Owners group.

We shall now create the tenant app catalog (if it does not already exist):

9. Open the SharePoint Admin center in your browser.
   

   https://mytenant-admin.sharepoint.us​



10. Select More Features in the left sidebar.


11. Locate the Apps section and click Open.
    

    https://mytenant-admin.sharepoint.us/_layouts/15/online/TenantAdminApps.aspx​



12. Select the App Catalog.


13. If you do not already have an app catalog, provide the following details:
    
    
    
    • Title: App Catalog
    
    
    • Web Site Address Suffix: preferred suffix for the app catalog, e.g. apps.
    
    
    • Administrator: SharePoint Administrator.
    
    
    
    

We shall now turn our attention to installing the latest version of PnP.PowerShell. At the time of writing this blog, the latest version of PnP.PowerShell is 1.5.x. Follow the instructions, in the box below, to install the latest pre-release version (required).

14. Open a new PowerShell console (v5.1 or Core 7.x).
    
    


15. Ensure the PnP.PowerShell module is loaded with the following:
    

    Import-Module -Name PnP.PowerShell

     

    
    


16. Run the following script ONCE per tenant to create an Azure App Registration for PnP:
    Note: Replace tenant with your tenant name.
    

    Register-PnPAzureADApp -ApplicationName "PnP PowerShell" `
    -Tenant [TENANT].onmicrosoft.us -Interactive `
    -AzureEnvironment USGovernmentHigh `
    -SharePointDelegatePermissions AllSites.FullControl User.Read.All

    Login with user credentials assigned Global Administrator role.
    If you previously registered PnP.PowerShell, check the App Registration in the Azure portal and make sure it has delegated permissions for AllSites.FullControl and User.Read.All.
    
    



17. Make a note of the GUID returned from step 4. This is the App/Client ID of the new PnP Azure App Registration.

Deploy Learning Pathways Template

Learning Pathways deploys as from a dedicated Look Book template. The following steps details downloading the template and deploying it via PnP.PowerShell.

18. Open a new PowerShell console (v5.1 or Core 7.x).


19. Ensure the PnP.PowerShell module is loaded with the following:
    

    Import-Module -Name PnP.PowerShell



20. Connect to your Learning Pathways site collection with the following:
    

    Connect-PnPOnline `
    -Url "Url of your Learning Pathways Site" `
    -AzureEnvironment USGovernmentHigh `
    -Interactive `
    -Tenant "[TENANT].onmicrosoft.us `
    -Client ID "Client ID from AAD app registration in Step #16"​



21. Enable custom scripts on your site with the following (note: check with you security team before enabling this feature):
    

    Set-PnPTenantSite -Identity "Url of your Learning Pathways Site" -DenyAddAndCustomizePages:$false​



22. Apply the template with the following:
    

    Invoke-PnPSiteTemplate -Path M365LP.pnp​



23. Connect the SharePoint Framework Web Part to your learning site with the following:
    

    Set-PnPStorageEntity `
      -Key MicrosoftCustomLearningSite `
      -Value "<URL of Learning Pathways site collection>" `
      -Description "Microsoft 365 learning pathways Site Collection";
    
    Set-PnPStorageEntity `
      -Key MicrosoftCustomLearningTelemetryOn `
      -Value $false `
      -Description "Microsoft 365 learning pathways Telemetry Setting";​

Public Preview of SAP NetWeaver, North Europe, OS, and new insights in Cluster Monitoring

by Contributed | Apr 14, 2021 | Technology

This article is contributed. See the original author and article here.

When customers move their mission critical SAP applications to Azure, a decision on which monitoring tool to use is carefully made. Various tools are evaluated, older pain points are reconsidered, and new cloud-specific monitoring needs are identified. Microsoft built Azure Monitor for SAP Solutions (AMS), currently in public preview, to address some of these needs. AMS is an Azure native monitoring solution that provides end-to-end SAP technical monitoring at one place in Azure portal.

AMS can be used by customers running their SAP workloads on Azure virtual machines as well as Azure large instances to monitor their SAP on Azure landscapes. Currently, telemetry collections for the following are supported: SAP HANA, Microsoft SQL Server and High-availability (pacemaker) clusters. And the following regions are supported: East US, East US 2, West US 2, West Europe.

Today I am excited to share the following capabilities in public preview:

• SAP NetWeaver (ABAP & JAVA)


• Operating system (SUSE & RHEL)


• Additional metrics for High-availability (pacemaker) cluster


• North Europe region

SAP NetWeaver data is collected from functions in SAP Start service which are provided on SAPControl SOAP Web Service. Following metrics are supported with this release: SAP instance availability, process availability, work process utilization, enqueue lock statistics, queue statics and trends. More to follow. Data for high-availability (pacemaker) clusters is collected from ha cluster exporter, which runs on every node in a cluster. Previously, statuses for nodes and resources were available, now customers can see information about location constraints, failure counts and trends on node and resource status. Operating systems (SUSE and RHEL) data is collected through node exporter which is required to run on each host. Metrics like:

• CPU usage by process


• Persistent memory


• Swap memory


• Memory usage


• Disk utilization,


• Network information

and more are available. OS monitoring in AMS is specifically useful for customers running workloads on Azure large instances (BareMetal).

Screenshots of visualizations from AMS

With this release, AMS provides end-to-end technical monitoring for all layers in SAP’s 3-tier architecture at one place. Customers can use AMS to visualize, monitor and troubleshoot all layers and visually corelate data across them. Further, customers can create Azure dashboards, with few clicks, to visualize SAP telemetry and non-SAP telemetry (other Azure services) in single-pane-of-glass. This can be done by combining telemetry from AMS and Azure monitor.

Since end-to-end technical monitoring is available within Azure portal, both BASIS administrators and infrastructure teams in customer’s organization can use AMS to monitor SAP on Azure. Moreover, Hosters/service integrations/partners can use AMS to monitor SAP systems for their customers and view cross-tenant SAP telemetry in Azure portal through integration between AMS and Azure Light house.

To get started, log into Azure portal and search ‘Azure Monitor for SAP Solutions’ in Azure Marketplace. Please see the links below for further information.

Share your thoughts with AMS product group: AMS asks & feedback form

Learn more:

• Step by step AMS onboarding


• SAP NetWeaver in AMS podcast


• AMS announcement


• AMS public documentation


• AMS quick start video


• AMS Azure Friday video


• AMS podcast


• AMS integration with Azure Lighthouse


• AMS & SUSE collaboration

 Other helpful links:

• SAP start service


• HA cluster exporter


• Node exporter

Get results from end-to-end training solutions offered by Microsoft Learning Partners

by Contributed | Apr 14, 2021 | Technology

This article is contributed. See the original author and article here.

This blog post is the second in a series of three that examines the results of a recent IDC study, Leveraging Microsoft Learning Partners for Innovation and Impact.* The first post, New study shows the value of Microsoft Learning Partners, provides a high-level look at the benefits of using a Learning Partner to meet your technical skilling needs.

We commissioned IDC researchers to explore the value of Microsoft Learning Partners, a worldwide network of authorized training providers that work with organizations to fill skill gaps and meet their strategic learning and business goals. In the first post, we explored how Microsoft Learning Partners can play a key role in delivering learning results that make a measurable difference in organizations’ cloud initiatives and digital transformation efforts.

Among the findings, four areas consistently stood out as essential capabilities. In this post, we dive into two of them—specifically, that organizations benefit from working with a Learning Partner which provides:

• An end-to-end solution that starts with consultation to identify needs and wraps up by evaluating the program’s success.


• Value-added services that support learners, such as hands-on labs and custom content.

The study identified two other top-of-mind areas that we’ll cover in our third post—the quality of the training content and delivery, and the flexibility, scale, and speed of the partner.

To learn more about all four qualities, download Top reasons to get IT training from a Microsoft Learning Partner.

Customized, end-to-end services help you assess gaps, simplify learning, and meet goals

Learning initiatives are usually complex, involving many people and moving parts. Rather than spin out various tasks to different providers, organizations can benefit from working with one provider that can deliver an end-to-end training solution.

Microsoft Learning Partners provide a range of capabilities that help enterprises build and execute successful training initiatives, including:

• Identifying an organization’s skill gaps. As consultants, Learning Partners work with their clients to offer proactive guidance and solutions. They work with organizations to identify the requirements and then align their services to meet a client’s specific needs, coordinating and delivering the resources that meet the objectives.


• Simplifying the learning initiative. By shifting the task to a Learning Partner, organizations can efficiently move their training programs forward. Learning Partners take care of the details, aligning the curriculum to the requirements, coordinating activities, and managing learning outcomes for their clients.

A mix of value-added services meet learners’ needs and deliver results

People learn best from a variety of tools and approaches, and flexibility is important, as Dan O’Brien notes. O’Brien is the President, United States and Canada, of Fast Lane, a worldwide provider of advanced IT education for leading technical vendors. “When we train [a large professional services firm], many of their staff are billable resources who can’t sit in a five-day class. We built a tailored training model specifically for this use case: We trained one day a week over five weeks, minimizing the impact on work performance.”*

Learning Partners make a difference in their scope of offerings, including hands-on labs, a mix of self-paced and instructor-led training, custom content, role-based learning paths, mentoring and discussion groups, and assessments before, during, and after the training. Services may also extend to preparation for certification exams.

Learning Partners give organizations the tools they need to successfully meet their business and learning goals through:

• Customized learning programs. Authorized Microsoft Learning Partners make training programs as relevant as possible for learners. They can use standard courses and learning paths, and they can tailor the content or flow to suit an organization’s unique situation.


• Training delivery aligned to client requirements. Some organizations want instructor-led training and boot camps, and some prefer e-learning. Learning Partners are experts at finding the right delivery model and experience for their customers.

Next step: Get the training program that’s right for you

Microsoft Learning Partners can help organizations get the most impact from their learning initiatives. The key is to find providers that offer end-to-end solutions and value-added services for learners of all types.

Find a Microsoft Learning Partner

Related posts

Sharpen your technical skills with instructor-led training

Leading Learning Partners Association—a unique organization for delivering Microsoft training

Need another reason to earn a Microsoft Certification?

Technical certifications could help drive business optimization