AKSe on Azure Stack Hub PNU process

by Scott Muniz | Jul 13, 2020 | Azure, Microsoft, Technology, Uncategorized

This article is contributed. See the original author and article here.

In our recently released AKS Engine on Azure Stack Hub pattern we’ve walked through the process of how to architect, design, and operate a highly available Kubernetes-based infrastructure on Azure Stack Hub. As production workloads are deployed, one of the topics that need to be clear and have operational procedures assigned, is the Patch and Update (PNU) process and the differences between Azure Kubernetes Service (AKS) clusters in Azure and AKS Engine based clusters on Azure Stack Hub. We have invited Heyko Oelrichs, who is a Microsoft Cloud Solution Architect, to explore these topics and help start the PnU strategy for the AKSe environments on Azure Stack Hub.

Before we start let’s introduce the relevant components: 

• Azure Kubernetes Service (AKS) is the Azure service that offers a managed Kubernetes service. With AKS, the customer gets a fully managed control plane while some components, such as worker nodes, have a shared responsibility. 

• AKS Engine is the open-source tool (hosted on GitHub) that is also used in Azure (under the covers) to deploy managed AKS clusters and is available to provision unmanaged Infrastructure-as-a-Service (IaaS) based Kubernetes Clusters in Azure and Azure Stack Hub.  
• Azure Stack Hub is an extension of Azure that provides Azure services in the customer’s or the service provider’s datacenter.  

The PNU process of a managed AKS cluster in Azure is partially automated and consists of two main areas: 

1. Kubernetes version upgrades are triggered manually either through the Portal, Azure CLI or ARM. These upgrades contain, next to the Kubernetes version upgrade itself, upgrades of the underlaying base OS image if available. These upgrades typically cause the reboot of the cluster nodes. 

   Our recommendation is to regularly upgrade the Kubernetes version in your AKS cluster to stay supported and current on new features and bug fixes.  

2. Security updates for the base OS image are applied automatically to the underlaying cluster nodes. These updates can include OS security fixes or kernel updates. AKS does not automatically reboot these Linux nodes to complete the update process. 

The PNU process on Azure Stack Hub is pretty much similar with a few small differences we want to highlight here. First thing to note is that Azure Stack Hub runs in a customer or service provider data center and is not managed or operated by Microsoft.  

That also means that Kubernetes clusters deployed using AKS Engine on Azure Stack Hub are not managed by Microsoft. Neither the worker nodes nor the control plane. Microsoft provides the tool AKS Engine and the base OS images (via the Azure Stack Hub Marketplace) you can use to manage and upgrade your cluster.  

On a high level, AKS Engine helps with the most important operations: 

• Deploy a new Kubernetes cluster 

• Scale an existing Kubernetes cluster 
• Upgrade to a newer Kubernetes and base OS image version 
• Upgrade the base OS image only 

Important to note though is, that AKS Engine allows you to upgrade only clusters that were originally deployed using the tool, clusters that were created without and outside of AKS Engine cannot be maintained and upgraded using AKS Engine.  

Upgrade to a newer Kubernetes version 

The aks-engine upgrade command updates the Kubernetes version and the AKS Base Image. Every time that you run the upgrade command, for every node of the cluster, the AKS engine creates a new VM using the AKS Base Image associated to the version of aks-engine used. 

The Azure Stack Hub Operator together with the Kubernetes Cluster administrator should make sure, prior to each upgrade: 

• that no system updates or scheduled tasks are planned 
• that the subscription has enough space for the entire process 
• that you have a backup cluster and that it is operational 

• that the required AKS Base image is available, the right AKS Engine version is used as well as that the target Kubernetes version is specified and supported 

The aks-engine repository on GitHub contains a detailed description of the upgrade process.  

Upgrade the base OS image only 

There might be valid reasons, for example dependencies to specific Kubernetes API versions and others, to not upgrade to a newer Kubernetes version, while still upgrading to a newer release of the underlaying base OS image. Newer base OS images contain the latest OS security fixes and kernel updates. This base OS image only upgrade is possible by explicitly specifying the target version, see here. 

The process is the same as for the Kubernetes version upgrade and also contains a reboot/recreation of the underlaying cluster nodes. 

Applying security updates 

The third area, that’s already baked into AKS Engine based Kubernetes clusters and does not need manual intervention is the process of how security updates are applied. This applies for example to Security updates that were released before a new base OS image is available in the Azure Stack Hub Marketplace or between two aks-engine upgrade runs, e.g. as part of a monthly maintenance task. 

These Security updates are automatically installed using the Unattended Upgrade mechanism. Unattended Upgrade is a tool built into Debian, which is the foundation of Ubuntu which is the Linux distro used for AKS and AKS Engine based Kubernetes clusters. It’s enabled by default and installs security updates automatically, but does not reboot the Kubernetes cluster nodes.  

Note: this automatic installation is done in connected environments, where the Azure Stack Hub workloads in user-subscriptions have access to the Internet. Disconnected environments need to follow a different approach. 

Rebooting the nodes can be automated using the open-source KUbernetes REboot Daemon (kured) that watches for Linux nodes that require a reboot, then automatically handle the rescheduling of running pods and node reboot process. 

Update types and components 

Conclusion and Responsibilities 

• New AKS Base OS Images are regularly released via the Azure Stack Hub Marketplace and have to be downloaded by the Azure Stack Operator. 
• New AKS Base OS Images and Kubernetes versions are applied using aks-engine upgrade and include the recreation of the nodes – this does not affect the operation of the cluster or the user workloads 
• Azure Stack Hub Operators play a crucial role in the overall upgrade process and should be consulted and involved in every upgrade process
• *very important* the Azure Stack Hub Operator should always consult the Release Notes that come with each update and inform the Kubernetes cluster administrator of any known issues. 
• Kubernetes cluster operators have to be aware of the availability of new updates for Kubernetes and AKS Engine and to apply them accordingly.  
• AKS Engine supports specific versions of Kubernetes and the AKS Base Image. 
• Security updates and kernel fixes are applied automatically and do not automatically reboot the cluster nodes. 
• Kubernetes cluster operators should implemented kured or other solutions to gracefully reboot cluster nodes with pending reboots to complete the update process 

This article and especially the list of responsibilities and considerations above is intended to give you a starting point and an idea of how to structure and execute the PNU process for AKSe environments. The details of the PNU process and how they relate to the application architecture are the most critical pieces of a successful and reliable operation. Separating the layers (the Azure Stack Hub platform, the ASKe platform, the application and respective data itself) would help towards being prepared to support an outage at each layer – and having operations prepared for each of them as well as mitigation steps required, would help minimize the risk. 

Johnson Controls makes working from home easier and more secure with Azure AD and Zscaler ZPA

by Scott Muniz | Jul 13, 2020 | Azure, Microsoft, Technology, Uncategorized

This article is contributed. See the original author and article here.

When it comes to remote work, the employee experience and security are equally important. Individuals need convenient access to apps to remain productive. Companies need to protect the organization from adversaries that target remote workers. Getting the balance right can be tricky, especially for entities that run hybrid environments. By implementing Zscaler Private Access (ZPA) and integrating it with Azure Active Directory (Azure AD), Johnson Controls was able to improve both security and the remote worker experience. In today’s “Voice of the Customer” blog, Dimitar Zlatarev, Sr. Manager, IAM Team, Johnson Controls, explains how it works.

Building a seamless and secure work-from-home experience

By Dimitar Zlatarev, Sr. Manager, IAM Team, Johnson Controls

When COVID-19 began to spread, because of our commitment to employee safety, Johnson Controls transitioned all our office workers to remote work. This immediately increased demand on our VPN, overwhelming the solution. Connections speeds slowed, making it difficult for employees to conveniently access on-premises apps. Some workers couldn’t connect to the VPN at all. To address this challenge, we deployed an integration between Azure AD and ZPA. In this blog, I’ll describe how ZPA and Azure AD support our Zero Trust journey, the roll-out process, and how the solution has improved the work-from-home experience.

Enabling productive collaboration in a dynamic, global company

Johnson Controls offers the world’s largest portfolio of building products, technologies, software, and services. Through a full range of systems and digital solutions, we make buildings smarter, transforming the environments where people live, work, learn and play. To support 105,000 employees around the world, Johnson Controls runs a hybrid technology environment. A series of mergers and acquisitions has resulted in over 4,000 on-premises applications for business-critical work. Some of these apps, like SAP, include multiple instances. Our strategy is to find software-as-a-service (SaaS) replacements for most of our on-premises apps, but in the meantime, employees need secure access to them. Before coronavirus shifted how we work, the small percentage of remote workers used our VPN with few issues.

To centralize authentication to our cloud apps, we use Azure AD. The system for cross-domain identity management (SCIM) makes it easy to provision accounts, so that employees can use single sign-on (SSO) to access Office 365 and non-Microsoft SaaS apps, like Workday, from anywhere.

We deployed Azure AD Self-Served Password Reset (SSPR) early in 2019 to allow employees to reset their passwords without helpdesk support. With this deployment, we’ve reduced helpdesk costs for password resets, account lockouts by 35% within the first three months and 50% a year later.

Securing mobile workers with a Zero Trust strategy

When employees began working from home, there were no issues accessing our Azure AD connected resources, but our VPN solution was significantly stretched. As an example, it could only support about 2,500 sessions in the entire continent of Europe, yet Slovakia alone has 1,700 employees. To expand capacity, we needed new equipment, but we were concerned that upgrading the VPN would be expensive and take too long. Instead, we saw an opportunity to accelerate our Zero Trust security strategy by deploying ZPA and integrating it with Azure AD.

Zero Trust is a security strategy that assumes all access requests—even those from inside the network—cannot be automatically trusted. In this model, we need tools that verify users and devices every time they attempt to communicate with our resources. We use Azure AD to validate identities with controls such as multi-factor authentication (MFA). MFA requires that users provide two authentication factors, making it more difficult for bad actors to compromise an account. Azure AD Privileged Identity Management (PIM) is another service that we use to provide time-based and approval-based role activation to mitigate the risks of unnecessary access permission on highly sensitive resources.

ZPA is a cloud-based solution that connects users to apps via a secure segment between individual devices and apps. Because apps are never exposed to the internet, they are invisible to unauthorized users. ZPA also doesn’t rely on physical or virtual appliances, so it’s much easier to stand up.

Enrolling 50,000 users in 3 weeks

To minimize disruption, we decided to roll out ZPA in stages. We began by generating a list of critical roles, such as finance and procurement, that needed to be enabled as quickly as possible. We then prioritized the remaining roles. This turned out to be the hardest part of the process.

Setting up ZPA with Azure AD was simple. First, Azure AD App Gallery enabled us to easily register the ZPA app. Then we set up provisioning, targeted groups, and then populated the groups. Once the appropriate apps were set up, we piloted the solution with ten users. The next day we rolled out to 100 more. As we initiated the solution, we worked with the communications team to let employees know what was happening. We also monitored the process. If there were issues with an app, we delayed deployment to the people with relevant job profiles. Zscaler joined our daily meetings and stood by our side throughout the roll out. By the end of the first week we had enabled 7,000 people. We jumped to 25,000 by the second, and by the third week 50,000 people were enrolled in ZPA.

Simplifying remote work with SSO

One reason the process went so smoothly is because the ZPA Azure AD integration is much easier to use than the VPN solution. Users just need to connect to ZPA. There is no separate sign-in. When employees learned how convenient it was, they asked to be enabled.

With ZPA and Azure AD, we were quickly able to scale up remote work. Employees are more productive with a reliable connection and simplified sign-in. And we are further down the path in our Zero Trust security strategy.

Learn more

In response to COVID-19, organizations around the world have accelerated modernization plans and rapidly deployed products to make work from home easier and more secure. Microsoft partners, like Zscaler, have helped many organizations overcome the challenges of remote work in a hybrid environment with solutions that integrate with Azure AD.

Learn how to integrate ZPA with Azure AD

Top 5 ways you Azure AD can help you enable remote work

Developing applications for secure remote work with Azure AD

Microsoft’s COVID-19 response