Invest in training and certification for your company’s success

by Contributed | Feb 17, 2021 | Technology

This article is contributed. See the original author and article here.

Imagine not doing so and your employees stay, say HSO managers

This post is based on a conversation with four people at HSO: Henk-Jan Brommer, Manager HSO Academy, which is Netherlands-based with a global reach; Rebecca Fox, Learning and Development Manager, HSO UK; Kristen Ramerini, Vice President, Human Resources, HSO US; and Kevin Moore, Human Resources Director, HSO Germany. In a spirit of collaboration, they stepped out of their daily tasks to reflect together on the challenges and achievements of the company’s journey to making training and certification integral to its organization, and they blended their voices into one representing the company’s passion and commitment.

Upskilling workers for the digital transformation of business is a necessity today. HSO, a global solution integrator with a hands-on attitude that facilitates companies’ digital transformation, knows this well, and it has established programs and processes to encourage and support tech learning. Training and certification in tech skills are integral to its success strategy—for its customers, its employees, and itself.

Founded in 1989, HSO built a track record helping enterprise systems adopt and use smart technology systems to empower their employees and become more efficient. In 2002, it moved to Microsoft Business Applications, first as a Dynamics ERP implementation partner and then adding Dynamics CRM, Analytics, Azure, Modern Workplace, Dynamics 365, and Microsoft Power Platform. Over the years, it has earned many gold and silver Microsoft awards, and it’s been an Inner Circle Partner for more than 10 years. In 2020, HSO won the Microsoft Partner of the Year award in the “Modernize Finance and Operations” category, and it was a finalist for Microsoft Partner of the Year in “Connected Field Service.”

To provide its customers with the best service, the company is committed to a “hands-on attitude.” Its consultants, who operate out of 27 offices around the world, are experienced solution architects and trusted advisors for enterprise clients across Dynamics 365 and other Microsoft platform technologies. Experts who specialize in global implementations, they work “up close and personal” with companies in a year-long dynamic process of analysis and design, matching the company’s business processes to Microsoft technologies, training the company to use the technologies, and supporting the implementation. That means HSO consultants need to know not just the customer but also which technologies will help that customer. In other words, they need soft skills and hard (or application) skills. “That’s why the company is committed to effective training and certification for all its consultants,” HSO managers say, “why we’re so passionate about it. By training our people effectively and supporting their technical knowledge with soft skills, we ensure that we have the best people to help our customers.”

A program and corporate academy to train and certify all employees

That commitment to training and certification is real. The company has always had a strong culture of learning and development. But, although its consultants always had the technical knowledge they needed for their work, they didn’t always get officially certified. Over the years, however, more and more customers requested that the HSO consultants working with them be certified. So, in 2019, when Microsoft initiated role-based certifications, HSO started a program to get every employee certified in Dynamics 365 and Microsoft Power Platform. Before this, many HSO consultants were getting certified in Dynamics 365, but there was no company policy supporting certification. After 2019, the company added certification as a primary focus alongside training. This is supported by its own corporate academy, HSO Academy, which was established in 2014, providing all employees with training and support for certification so they “are able to work for HSO with pleasure and with the right knowledge.”

The program and Academy started with the Dynamics 365 Fundamentals certification. The biggest challenge with that first wave of company-wide fundamentals certification, the managers say, was streamlining the training and certification for such a large number of people—more than a thousand. At one point, HSO Academy was delivering a Dynamics 365 Fundamentals training nearly every day of the week.

As Microsoft continued to update and add new certifications in Dynamics 365 and Microsoft Power Platform, HSO followed suit. For example, the company will be supporting the two new Dynamics 365 fundamentals certifications that in February 2021 will replace the Dynamics 365 Fundamentals certification it taught in its first certification push: one in Customer Engagement Apps (CRM) and one in Finance and Operations Apps (ERP). For all of its training, it uses the learning material on Microsoft Learn, plus Microsoft Official Curriculum. To this it adds its own training in soft business skills, including communication, handling different situations, and being a role model.

HSO intentionally incorporates training and support for certification for all its employees—all the way from consultants to noncustomer-facing employees and from new hires to experienced consultants.

Masterclass for young professionals and onboarding for all new hires

As part of HSO Academy, HSO offers a five-week training program for every young professional who is new to HSO: the Masterclass. The Masterclass, led by Henk-Jan Brommer, a Microsoft Certified Trainer (MCT), quickly prepares trainees for their role at HSO—from consultancy skills to application skills in the context of the digital transformation of business. “Even if they don’t have an IT background,” Brommer says, “we make sure they’re prepped for their first assignment with the customer in five weeks. Specifically, we train them in our HSO-led Dynamics 365 and HSO-led Power BI courses, with the goal of earning Microsoft Certification in Dynamics 365 for Finance and Operations or Dynamics 365 Customer Engagement by the end of the class. After a year, they’ve often earned two or more additional certifications.”

The practical examples and situations of the Masterclass keep the training hands-on rather than theoretical, and they help stimulate group dynamics. An added value to the Masterclass, Brommer notes, is that participants come from all over the world. In the process of learning together, they form “friendships for life” that carry over into their later work for the company’s customers, because they can more easily share their knowledge and experience with one another. This bond is nurtured after the Masterclass ends, when participants are placed on a team that works with a mentor or senior consultant on a client project, with the aim of running a client project as an independent junior consultant by the end of one year.

All new hires—junior or senior, tech-experienced or not—are encouraged and supported to get trained and certified. “In Germany, for example,” Moore explains, “HSO has an onboarding program in which we look at new hires’ career level and experience and determine with them which certifications they have to do, at which level.” Then, in the first six months, in addition to having a mentor, they have two or three meetings with HR to make sure they’re getting what they need, to see whether there are any roadblocks. “We want to make sure the people we hire are successful, that they’re growing, not stagnating,” the managers point out. “Competence leads to confidence. When people have knowledge, the skills they need, they’re better able to work independently and think outside the box. They can be creative and ask, How can I use Microsoft Power Platform to solve this problem? How can I make this solution work better? They have fun in their projects, and when you have fun, you have more success.”

Investing in people

The company’s commitment to training and certification is appreciated by HSO’s customers, and it has also resulted in greater acknowledgement and deeper partnership with Microsoft. But it’s much more than that, these managers say. For HSO, training and certification are “a way of investing in people.” When a person leaves the company, they take their knowledge, experience, and certifications with them. This doesn’t hold HSO back, however, because “the skills and knowledge of our employees are what differentiate us from the competition.”

“For our HSO employees, as well as for candidates during the recruiting process, one of the major factors in choosing HSO and remaining loyal to the company is our continuous learning and development culture. We see it this way: if we can enable our colleagues through trainings and certifications, they will be successful. And when they are successful, HSO is successful. This is a win-win-win situation for the employee, for HSO, and for our customers!”

When asked, “What is the value of training and certification for your company?,” the group shared this story that “says it all”: “A CFO says to a CEO: ‘I see that many people are being trained. Do you have any clue what that costs? What if they leave?’ To which the CEO replies: ‘Imagine not training them and they stay…’”

This investment in people is built into the structure of the company. Certification is a fully integrated part of the company’s Career and Performance Development Plan and each employee’s Personal Development Plan. In fact, it’s at the top of the list. Every year, managers work with each employee to choose which certification best suits them and together they make a plan to train for and earn that certification. They then follow up with regular conversations throughout the year to help them achieve their goals. “There’s a direct correlation between the level and type of certification and the career level of our people to fulfill their respective roles,” the managers note.

The fact that more and more Microsoft Certifications are becoming available makes it easier to tailor certification for employees. “That’s what we’re most enthusiastic about—the move to role-based certifications, the addition of so many more certifications, and the massive improvements in Microsoft Learn to support those certifications. Our options are no longer limited. It’s like we now have this candy store to pick and choose from. And Microsoft Learn is now an efficient one-stop shop for learning, whereas before we had to scrape together bits and pieces to get adequate training.”

This encouragement and support to learn and grow extends to employees who work in noncustomer-facing areas, such as Rebecca Fox, who works in Learning and Development, and Kristen Ramerini and Kevin Moore, who work in Human Resources. Ramerini and Moore are planning to work toward one of the new Dynamics 365 fundamentals certifications soon. Fox just passed Exam MB-901: Microsoft Dynamics 365 Fundamentals to earn her first certification. “If I can do it, anyone can!” she says. ”After I passed, I got lots of emails saying ‘Well done! You inspired me to get going on one.’”

Fox particularly appreciated the blended approach to learning of the virtual instructor-led training (VILT) that prepared her to take her certification exam. An instructor can talk people through the content, apply it to a specific area, and respond to questions. That’s what made it so exciting for her. The combination of Microsoft Learn content with engaging virtual instruction from a trainer is a winning one, Brommer agrees. Microsoft is on the right track with instructor-led training for exam preparation, he says, because it enables instructors—like those at HSO Academy—to take the Microsoft Learn building blocks and provide that “last mile” of a practical approach, tailoring the instruction to very specific areas, such as finance for retail or finance for the public sector. Not all companies can invest in their own in-house instructors and training like HSO, however, and that’s where Microsoft Learning Partners come in. Learning Partners offer exam prep courses led by MCTs. Companies can support individual employees in taking VILT training from Learning Partners and even arrange for custom trainings for their teams or groups.

With the support of the Academy, HSO also runs two-day regional training conferences for its employees. In the United Kingdom, for example, it runs a conference every 18 months for all employees in the region, consultant or not, so everyone has access to the Academy. Trainers teach the soft skills and bring in Brommer and others to help people learn the skills needed to pass certification exams, especially in Dynamics 365 or Microsoft Power Platform fundamentals. Another advantage of the Academy is that as an entity independent of a region it can bring together people from the company’s different regions and build learning communities in which people can share their knowledge, support one another in earning certifications, and nurture close bonds. “Creating learning communities,” these managers say, “is one of the things we’d advise other partners to consider.”

Employees who are trained, certified, and invested in learning and growing

Currently, the vast majority of HSO employees are certified in Dynamics 365, and the number certified in Microsoft Power Platform is increasing fast, especially because Microsoft Power Platform is one of the company’s main areas of investment for 2021 and beyond. Before, when doing its fit-gap analysis, the organization would offer customization as an option. With Microsoft Power Platform, customization has become the last option, because it offers so many more possibilities to fit business processes into the applications that the company recommends. Fortunately, this group says, HSO employees feel not only challenged but also encouraged and supported by this emphasis on certification. “Because our company culture is so focused on learning and development,” Ramerini explains, “we tend to attract people who are also invested in learning and growing. So there’s a natural flow. Our employees are proactive, and they work independently toward their goals, with our support.”

The company does try to add an element of fun to upskilling and certification by making it a friendly game. It sponsors games like “Who will be the first person with four different Dynamics 365 Fundamentals certificates?” or “Who will be the first person to get certified in two different areas?” Recently, it held a competition to use Microsoft Power Platform to create an app that would be useful in the HSO organization, with a prize of £5,000. “People took the initiative to upskill themselves using Microsoft Learn just to be able to participate,” the managers report. A number of the apps that employees created for the company are being used for HSO’s customers too, such as the Workplace Wizard, which creates a map of a workplace that calculates and displays social-distancing possibilities for COVID-19 on any particular day.

Clearly HSO has created a corporate culture and structured its organization to enhance learning and development. When asked what advice they would give other Microsoft partners about helping their employees get trained and certified in Dynamics 365 and Microsoft Power Platform, these managers replied: “Rewarding and recognizing employees for their efforts in getting trained on new technologies and obtaining the corresponding certifications is very important. We’ve found that rewarding team members with exam bonuses helps, as does recognizing people’s efforts multiple times and in many ways, for instance by posting announcements on Yammer, offering congratulations on earning certification in team and company meetings, and displaying achievements on an internal dashboard so everyone is aware of it. We recommend that other partners consider doing these things—and coming up with more ways.”

HSO’s commitment to training and certification is part of its strategy to help businesses future-proof themselves by digital transformation and to help their employees future-proof their careers by acquiring and validating the latest tech skills for business. So what is HSO planning for its own future? “We aim to be the leading global provider of technology-driven business solutions that improve the performance and results of our clients.” How does it plan to get there? “We plan to accelerate what we’re doing to keep up with and keep ahead of rapid changes. We want to keep moving forward and quickly adopt new Microsoft Certifications as soon as they become available. We’ve even offered to assist Microsoft in setting up additional certifications and helping to create content for that. Partnership in learning is the way forward.”

SAP ASCS HA Cluster (in Linux OS) failover to DR region using Azure Site Recovery

by Contributed | Feb 17, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

This blog provides guidance to perform the steps during the failover of SAP ASCS/ERS HA VMs in Linux cluster to the DR region in Azure using ASR. We will have details of changes to be made in the DR environment to re-configure the pacemaker cluster to start the ASCS/ERS HA environment with Azure Fence agent as STONITH device. Steps cover both SUSE Linux and RHEL OS. In SUSE Pacemaker cluster, we can also use SBD device (in place of Azure Fence agent) for fencing which requires additional VMs and its DR setup will require additional changes which is not covered in this blog.

Note : The specific procedures described have been exercised with these OS releases
• OS release #1 : SUSE Linux 12 SP5
• OS release #2 : RHEL 8.1
Please note that the procedures described have not been coordinated with the OS providers and therefore might not work in completeness with your specific implementations or with future OS releases. As a result you should test the procedures described thoroughly in your environment.

Also note that the procedure as described works only with Azure Fencing Agent and not with iSCSI SBD devices.

Disaster Recovery Architecture for SAP ASCS HA Cluster

The SAP ASCS/ERS HA cluster design in the primary and DR region in Azure is as described in the diagram and can be used as reference architecture for SAP HA & DR setup in Azure. Highly Available NFS File share to be used for common file systems of SAP. Azure Site Recovery (ASR) is recommended to be used for across region replication of the VMs for DR setup. An NFS fileshare needs to be available in the respective region for starting the SAP ASCS/ERS application services and should be synchronized between region for availability of latest data.

Preparations

• 
  

  Configure ASR for both the nodes of ASCS in the primary region.

  
  

1. Deploy the Resource Group, VNet, Subnet and Recovery Vault in the Secondary Region.


2. Click on the ‘Disaster recovery’ for the ASCS/ERS VMs. Select the DR region (e.g. West US 2).


3. In advanced settings, Select the target Resource Group, Vnet, Recovery vault, AV Set(if needed), PPG (if needed) and disks to be included.


4. Review the settings and start the Replication.

• 
  

  Check that ASR replication is 100% and its healthy.

  
  

• 
  

  Deploy Azure ILB for ASCS & ERS in DR region.

  
  

Define frontend IP, backend pool, Probe port and loadbalancing rules. Frontend IP would be different in the DR region. Probe port can be same as primary region ASCS/ERS cluster.

• 
  

  NFS files shares synchronization

  
  

NFS Fileshare for ‘sapmnt’, ‘trans’ and ‘usr/sap’ must be must be synchronized with Primary Region and available/mounted in the DR region. New location/path of NFS files needs to be updated in ‘/etc/fstab’ and cluster configuration the DR ASCS VMs.

Note: One of the options for NFS FileShare is to use Azure File NFS. As ASR can’t replicate NFS sources, one of the methods to replicate is to Copy the data to locally attached disk in the ASCS/ERS VMs using cronjob(for frequent interval copy)  so that ASR can replicate the data to DR region. Detailed steps are described in Appendix.

ASCS/ERS DR Failover

The following items are prefixed with either [A – DR] – applicable to all nodes of DR ASCS/ERS, [1-DR] – only applicable to node 1 of DR ASCS/ERS or [2-DR] – only applicable to node 2 of DR ASCS/ERS

1. Perform the ‘Failover’ OR ‘Test Failover’ of ASCS/ERS Cluster VMs using ASR to the DR region.

2. [A – DR] Update the IP addresses of the VMs and virtual IPs either in AD/DNS or in ‘hosts’ file.


3. [A – DR] Mount the NFS filesystems for ‘sapmnt’, ‘trans’ and ‘SYS’. Mounting process depends on the NFS Share type (ANF / Azure Files NFS(in preview as of February 2021)).


4. [A – DR] Ensure that contents of ‘sapmnt’, ‘trans’ and ‘SYS’ filesystems are synchronized from Primary Region.


5. [A – DR] Update the VMs physical IP addresses in /etc/corosync/corosync.conf

nodelist {

        node {

                ring0_addr: 173.30.0.61

                nodeid: 1

        }

        node {

                ring0_addr: 172.30.0.62

                nodeid: 2

        }

Note: This step is only required in SUSE Linux.

6. [A – DR] Start the pacemaker cluster using the command.
   
   
   
   • For SUSE Linux
   
   
   
   

systemctl start pacemaker

6. 
   
   
   
   • For RHEL
   
   
   
   

pcs cluster start

7. [A – DR] Keep the cluster in maintenance mode.  
   
   
   
   • For SUSE Linux                                                                                                                   
   
   
   
   

sudo crm configure property maintenance-mode=”true”

7. 
   
   
   
   • For RHEL
   
   
   
   

sudo pcs property set maintenance-mode=true

8. [1-DR] Update the pacemaker configuration and save the changes. 

For SUSE Linux : The properties of the resources can be changed in the GUI tool ‘Hawk’ (https://<hostname>:7630/) or using the syntax “crm configure edit” (use ‘vi’ editor commands to update the content)

For RHEL : The properties of the resources can be changed using the ‘PCSD web UI’ (https://<hostname>:2224/). Once you start the pcs web UI, click on ‘+Add Existing’ and enter hostname of the cluster to see the properties.

• Fileshare location of ‘ASCS’ and ‘ERS’.

• Probe Port numbers of ILB for ASCS and ERS (if different probe port numbers are used in Primary and DR)


• Frontend IP (virtual IP) defined in ILB for ASCS and ERS.

• Azure Fence Agent.
  
  
  
  • We can reuse the Azure Fence agent API created for ASCS/ERS cluster(in the primary region) in the DR region. Optionally, we can create a new Azure Fence Agent API.
  
  
  • Assign the custom role to the Service Principle for the DR VMs as per the link.
  
  
  • Update the Azure Fence agent details (new resource group) in the cluster configuration.
  
  
  
  

Note : Azure Fence Agent requires outbound connectivity to public end points as documented, along with possible solutions, in Public endpoint connectivity for VMs using standard ILB.

Note : While performing ‘Test Failover’ in ASR, VM name created in the DR Region will be suffixed by ‘-test’ but hostname at operating system will be same as Primary Region VMs. Since VM name doesn’t match with node name(hostname) in the cluster, we need to add parameter ‘pcmk_host_map’ and map hostname & VM name in Azure Fence Agent configuration in the pacemaker. This will ensure fencing of the VM during cluster testing.

9. [A – DR] Ensure that ‘ASCS<nr>’ and ‘ERS<nr>’ filesystems contents are synchronized with the data from Primary region.


10. [1-DR] Remove the maintenance mode and cleanup cluster resources (if required).
    
    
    
    • For SUSE Linux
    
    
    
    

sudo crm configure property maintenance-mode=”false”

9. 
   
   
   
   • For RHEL
   
   
   
   

sudo pcs property set maintenance-mode=false

11. Check the cluster status.

12. Continue with the DR activation tasks for DB and application servers.


13. Perform the DR validation tasks and cluster testing in the DR environment.


14. Once DR test is completed, ‘Cleanup test failover’ in ASR for both ASCS/ERS VMs.

Appendix

This section describes steps to synchronize Azure Files NFS between primary and secondary region. This method of synchronization is one of the several possible ways to achieve data synchronization.

To setup ASCS/ERS cluster with Azure Files NFS(in public preview as of February 2021), please refer to the blog.

High level steps

• Attach and Mount Azure premium disks to the VMs in the primary region ASCS/ERS VMs.


• Regularly Copy the NFS share data/files into an azure disk using cronjob script.


• ASR will be able copy Azure Disk to DR region. Ensure this disk included in the ASR replication.


• During DR activation, Once the VMs are available, mount the Azure Files NFS from the DR region.


• Copy the data/files from local disks to Azure Files NFS mount points.

Detailed Steps:

Provided steps as reference by considering SAP SID as T01, ASCS system number as ‘00’ and ERS system number as ‘02’.

In Primary Region

1. [A] Add azure premium disk to both of VMs of ASCS/ERS cluster and mount the filesystem (e.g. /sapfoldercopy ).


2. [A] Create folders in the filesystem.

sudo mkdir -p /sapfoldercopy/T01ASCS00

sudo mkdir -p /sapfoldercopy/T01ERS02

sudo mkdir -p /sapfoldercopy/sapmntT01

sudo mkdir -p /sapfoldercopy/trans

sudo mkdir -p /sapfoldercopy/usrsapT01

chown <sid>adm:sapsys /sapfoldercopy/*

3. [A] Create shell script to copy data from NFS fileshare to local azure disk.

>>vi copy_sap_folders.sh

#!/bin/sh

cp -p -u -R /sapmnt/T01/ /sapfoldercopy/sapmntT01/

cp -p -u -R /usr/sap/trans/ /sapfoldercopy/trans/

cp -p -u -R /usr/sap/T01/ /sapfoldercopy/usrsapT01/

erscount=”$(ls -l /usr/sap/T01/ERS02/ | wc -l)”

if [[ $erscount -gt 1 ]]

then

{

        cp -p -R /usr/sap/T01/ERS02/ /sapfoldercopy/T01ERS02/

        mv /sapfoldercopy/T01ASCS00/ASCS00 /sapfoldercopy/T01ASCS00/ASCS00_old

}

fi

ascscount=”$(ls -l /usr/sap/T01/ASCS00/ | wc -l)”

if [[ $ascscount -gt 1 ]]

then

{

        cp -p -R /usr/sap/T01/ASCS00/ /sapfoldercopy/T01ASCS00/

        mv /sapfoldercopy/T01ERS02/ERS02 /sapfoldercopy/T01ERS02/ERS02_old

}

fi

Note : comment out copy of ‘sapmnt’, ‘trans’ and ‘usrsapT01’ in one of the VM as contents are same in both the VMs.

4. [A] Ensure file have right ownership and permissions.

chown <sid>adm:sapsys copy_sap_folders.sh

chmod 755 copy_sap_folders.sh

5. [A] Schedule the cronjob for use <sid>adm

>>crontab -e 

15,30,45,59 * * * * /home/t01adm/copy_sap_folders.sh

In Secondary Region, during DR activation OR DR testing

1. [A – DR] Update the /etc/fstab files to mount the Azure Files NFS in the secondary region.

>> mount -a

2. [1-DR]Update the cluster configuration to update the Azure Files location for ASCS00 and ERS02 folders. Details are described in the main section of this document.


3. [A – DR] Copy the contents from Azure local disk filesystem (/sapfoldercopy) to Azure Files NFS filesystem paths in respective locations.

Innovation without Authority – M365 for Government February User Group

by Contributed | Feb 17, 2021 | Technology

This article is contributed. See the original author and article here.

The stereotypes of burdensome bureaucratic processes and legacy mindsets are self-fulfilling prophecies robbing municipalities and institutions of the change-makers we need to effect positive change at scale.

Listen to Sean Bugler, a public sector cloud architect, as he discusses how he’s using Power Automate, Power Apps, and SharePoint to empower an entirely new class of power users to reimagine what workflows can (and should) be in 2021 and beyond.

YouTube Recording: https://youtu.be/Vj9jGI9lVuU

https://www.youtube-nocookie.com/embed/Vj9jGI9lVuU

LinkedIn Group: Microsoft 365 for Government DC Users Group | Groups | LinkedIn

Relevant links

Sean’s social:
:cloud: Sean Bugler | LinkedIn
https://twitter.com/sbglr

Sean Bugler (@sbglr) / Twitter

Sean Bugler and Matt Wade’s Weekly Microsoft News Show:
https://www.youtube.com/playlist?list=PLF_VzpIgKVtxbA93KO39YwcTo7CbuoaMa

Sean’s journey into IT:
The Geek Within Ep9: Sean Bugler – YouTube

Innovation Backlog (mentioned during presentation):
https://docs.microsoft.com/en-us/power-platform/guidance/coe/setup-innovationbacklog#:~:text=The%20Innovation%20Backlog%20solution%20contains%20assets%20that%20are,installed%20in%20Production%20or%20Dataverse%20for%20Teams%20environments.

Manage Certificates on your Hybrid Servers using Azure Arc Key Vault Extension

by Contributed | Feb 17, 2021 | Technology

This article is contributed. See the original author and article here.

Managing certificates is an important scenario when it comes to server management. You want to make sure you can roll out certificates to your servers and manage these from a central place. In a pure on-premises environment, we have done this for example by using Group Policies (GPOs). But if you want to run this in a hybrid cloud environment, where machines are not only in your own datacenter but also running at different edge locations or even other cloud providers, where machines in some cases are not even are domain-joined? Here is where Azure Arc can help you with, Azure Arc enabled servers allows you to manage your Linux and Windows Servers directly from the Azure control plane, and with the new Azure Arc Key Vault extension, you can also manage certificates on these machines.

Since Azure Arc enabled Servers have a managed identity, you can use that managed identity not just for your applications, but also for extensions and accessing Azure Key Vault.

Instead of individually copying the certificate to each machine, the PKI admin just has to upload or generate certificates in a Key Vault and configure which servers are allowed to download them. Then, the server admin deploys the Azure Key Vault extension to their servers (the same extension works for both Azure and Arc enabled servers) and specifies which certificates should be installed on the server, and how frequently the server should check for updates. From there, the extension takes care of the rest. It uses the unique managed identity assigned to every Arc enabled server to authenticate to Azure Key Vault and download the certificates. When it comes time to renew a certificate, the PKI admin only needs to update the copy in Key Vault. The extension will take care of downloading it to each server automatically.

Azure Arc Enabled Server Key Vault Extension

Getting started with the Azure Key Vault extension for Arc enabled servers

This extension is currently in preview, and you can find more information about it here in the announcement blog post by Ryan Puffer (Program Manager for Azure Arc enabled server).

Prerequisites 

To get started, you will need the following:

• An Azure Arc enabled server running supported versions of Linux or Windows server. You can follow these simple steps to add a server to Azure Arc.


• An Azure Key Vault with at least one certificate. If you want to quickly create a certificate in Azure Key Vault, check out the following tutorial on Microsoft Docs.
  Certificate in Azure Key Vault
  

   

  
  


• To deploy the extension you will need the Azure Connected Machine PowerShell module (Az.ConnectedMachine) which you can run and install on your local admin machine or in Azure Cloud Shell by using the following command:
  

  Install-Module Az.ConnectedMachine

Set up and deploy the Key Vault extension to Azure Arc

Set permission to the Key Vault so the Arc enabled server has a system-assigned managed identity that can access it. 

You can configure permissions on your vault by going to it in the Azure Portal, clicking Access policies in the navigation pane, and then Add Access Policy. In the Secret permissions drop down, tick the boxes for Get and List. Then, next to Select Principal, click None selected to open the AAD object picker. Search for your Arc enabled server by its name, click it, then click Select. Click Add to finish configuring the Arc enabled server’s permissions then click Save to commit the change.

If you’re using the Azure Key Vault RBAC, grant the Arc enabled server the Key Vault Secrets User role in Access control (IAM) for the vault.

Deploy the Azure Arc Key Vault extension

Now you can deploy the extension to the server. For that run this command on your admin workstation with Azure PowerShell or Azure Cloud Shell and the Az.ConnectedMachine module installed.

$Settings = @{
  secretsManagementSettings = @{
    observedCertificates = @(
      "https://YOURVAULTNAME.vault.azure.net/secrets/YOURCERTIFICATENAME"
      # Add more here in a comma separated list
    )
    certificateStoreLocation = "LocalMachine"
    certificateStoreName = "My"
    pollingIntervalInS = "3600" # every hour
  }
  authenticationSettings = @{
    # Don't change this line, it's required for Arc enabled servers
    msiEndpoint = "http://localhost:40342/metadata/identity"
  }
}

$ResourceGroup = "ARC_SERVER_RG_NAME"
$ArcMachineName = "ARC_SERVER_NAME"
$Location = "ARC_SERVER_LOCATION (e.g. eastus2)"

New-AzConnectedMachineExtension -ResourceGroupName $ResourceGroup -MachineName $ArcMachineName -Name "KeyVaultForWindows" -Location $Location -Publisher "Microsoft.Azure.KeyVault" -ExtensionType "KeyVaultForWindows" -Setting (ConvertTo-Json $Settings)

In my case this looked something like this:

$Settings = @{
  secretsManagementSettings = @{
    observedCertificates = @(
      "https://toms-awesomearc-keyvault.vault.azure.net/secrets/TomsAwesomeCert"
      # Add more here in a comma separated list
    )
    certificateStoreLocation = "LocalMachine"
    certificateStoreName = "My"
    pollingIntervalInS = "3600" # every hour
  }
  authenticationSettings = @{
    # Don't change this line, it's required for Arc enabled servers
    msiEndpoint = "http://localhost:40342/metadata/identity"
  }
}

$ResourceGroup = "toms-azurearcservers-rg"
$ArcMachineName = "TOMSVM"
$Location = "westeurope"

New-AzConnectedMachineExtension -ResourceGroupName $ResourceGroup -MachineName $ArcMachineName -Name "KeyVaultForWindows" -Location $Location -Publisher "Microsoft.Azure.KeyVault" -ExtensionType "KeyVaultForWindows" -Setting (ConvertTo-Json $Settings)

When the extension has finished installing you should see your certificate on your Azure Arc enabled server.

Certificate on local machine deployed by Azure Arc Key Vault extension

For Linux machines you can run the following to deploy the extension:

$Settings = @{
  secretsManagementSettings = @{
    observedCertificates = @(
      "https://YOURVAULTNAME.vault.azure.net/secrets/YOURCERTIFICATENAME"
      # Add more here, don't forget a comma on the preceding line
    )
    # The cert store location is optional, the default path is shown below
    # certificateStoreLocation = "/var/lib/waagent/Microsoft.Azure.KeyVault.Store/"
    pollingIntervalInS = "3600" # every hour
  }
  authenticationSettings = @{
    msiEndpoint = "http://localhost:40342/metadata/identity"
  }
}

$ResourceGroup = "ARC_SERVER_RESOURCE_GROUP_NAME"
$ArcMachineName = "ARC_SERVER_NAME"
$Location = "ARC_SERVER_LOCATION (e.g. eastus2)"

New-AzConnectedMachineExtension -ResourceGroupName $ResourceGroup -MachineName $ArcMachineName -Name "KeyVaultForLinux" -Location $Location -Publisher "Microsoft.Azure.KeyVault" -ExtensionType "KeyVaultForLinux" -Setting (ConvertTo-Json $Settings)

Conclusion

Azure Arc enabled servers is a great way to manage your servers in a hybrid and multi-cloud environment. With the new Key Vault extensions (preview) Azure Arc makes it easy to manage certificates on servers you need to manage, where every they are deployed in a secure way.

If you want to learn more check out the following links:

• Azure Key Vault Extension for Windows / Linux


• Azure Key Vault


• Azure Arc enabled servers


• Have feedback on the AKV extension? Share it with us on UserVoice!

You can also watch the ITOps Talks All Thing Hybrid, where I had the chance to talk with Ryan about Azure Arc enabled Servers.

If you have any questions feel free to leave a comment.