This article is contributed. See the original author and article here.
In general, the recommendation is to use Automatic OS upgrade feature of Virtual Machine scale set as patching solution for Service Fabric in production refer to (it needs durability of Silver and above for nodetype) :
However in this approach updates can happen anytime (but will be rolling upgrade) i.e. when new images are published. If you don’t want this and need more control like schedule patching during non-peak time you can consider Patch Orchestration Application refer https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-patch-orchestration-application. Otherwise if you need total control like want to test updates in lower environments and then only patch prod, then you have to manually upgrade images refer : https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-automatic-upgrade#manually-trigger-os-image-upgrades or simply disable nodes one by one with intent restart and then update Windows and then enable node again.
Important Notes:
- Ideally, when “enableAutomaticUpdates” is set to True, you are enabling windows updates i.e. patch upgrades, etc. (not the upgrade from 2012 to 2016). By default its True. These updates doesn’t happen in rolling fashion.
- For scale sets using Windows virtual machines using automatic OS upgrade feature i.e enableAutomaticOSUpgrade set to True, starting with Compute API version 2019-03-01, the property virtualMachineProfile.osProfile.windowsConfiguration.enableAutomaticUpdates property must set to false in the scale set model definition. The above property enables in-VM upgrades where “Windows Update” applies operating system patches without replacing the OS disk. With automatic OS image upgrades enabled on your scale set, an additional update through “Windows Update” is not required. So if your using any patching solution in prod, Automatic OS Upgrade feature / Patch Orchestration Application / Manual OS upgrades, ideally you should be set enableAutomaticUpdates to false.
If you are patching VMSS nodes you should also make sure the windows container which is running in VMSS nodes is patched and the windows version should be matched with VMSS node and the container.
In windows containers, its recommended that both should be patched to latest however host images using 1809 and above does not need to have matching revisions or if you are using Hyper-V isolation mode. Refer to examples. You can also refer to https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/update-containers#how-to-get-windows-server-container-updates for getting Container updates.
Windows Server containers currently don’t support scenarios where Windows Server 2016-based containers run in a system where the revision numbers of the container host and the container image are different. For example, if the container host is version 10.0.14393.1914 (Windows Server 2016 with KB4051033 applied) and the container image is version 10.0.14393.1944 (Windows Server 2016 with KB4053579 applied), then the image might not start.
However, for hosts or images using Windows Server version 1809 and later, this rule doesn’t apply, and the host and container image don’t need to have matching revisions.
We recommend you keep your systems (host and container) up-to-date with the latest patches and updates to stay secure.
Example 1: The container host is running Windows Server 2016 with KB4041691 applied. Any Windows Server container deployed to this host must be based on the version 10.0.14393.1770 container base images. If you apply KB4053579 to the host container, you must also update the images to make sure the host container supports them.
Example 2: The container host is running Windows Server version 1809 with KB4534273 applied. Any Windows Server container deployed to this host must be based on a Windows Server version 1809 (10.0.17763) container base image, but doesn’t need to match the host KB. If KB4534273 is applied to the host, the container images will still be supported, but we recommend you update them to address any potential security issues.
Container Patching
In simple term in your case you have to update your docker file, working with containers is not the same as working with real servers or VM’s you support for months or years. A container image is a static snapshot of the filesystem (and Windows registry and so on) at a given time.
Container images have layers
First have a look how a container image looks like. It is not just a snapshot. A container image consist of multiple layers. When you look at your Dockerfile you normally use a line like FROM microsoft/windowsservercore.
Your container image then uses the Windows base image that contains a layer with all the files needed to run Windows containers.
If you have some higher level application you may use other prebuilt container images like FROM microsoft/iis or FROM microsoft/aspnet. These images also re-use the FROM microsoft/windowsservercore as base image.
On top of that you build your own application image with your code and content needed to run the application in a self contained Windows container.
Behind the scenes your application image now uses several layers that will be downloaded from the Docker Hub or any other container registry. Same layers can be re-used for different other images. If you build multiple ASP.NET applications as Docker images they will re-use the same layers below.
But now back to our first question: How to apply Windows Updates in a container image?
The Windows base images
Let’s have a closer look at the Windows base images. Microsoft provides two base images: windowsservercore and nanoserver. Both base images are updated on a regular basis to roll out all security fixes and bug fixes. You might know that the base image for windowsservercore is about 4 to 5 GByte to download.
So do we have to download the whole base image each time for each update?
If we look closer how the base images are built we see that they contain two layers: One big base layer that will be used for a longer period of time. And there is a smaller update layer that contains only the patched and updated files for the new release.
So updating to a newer Windows base image version isn’t painful as only the update layer must be pulled from the Docker Hub.
But in the long term it does not make sense to stick forever to the old base layer. Security scanners will mark them as vulnerable and also all the images that are built from them. And the update layer will increase in size for each new release. So from time to time there is a “breaking” change that replaces the base layer and a new base layer will be used for upcoming releases. We have seen that with the latest release in December.
From time to time you will have to download the big new base layer which is about 4 GByte for windowsservercore (and only about 240 MByte for nanoserver, so try to use nanoserver whereever you can) when you want to use the latest Windows image release.
Keep or Update ? Should I avoid updating the Windows image to revision 576 to keep my downloads small? No!
Recommendation is to update all your Windows container images and rebuild them with the newest Windows image. You have to download that bigger base layer also only once and all your container images will re-use it.
Perhaps your application code also has some updates you want to ship. It’s a good time to ship it on top of the newest Windows base image. So is recommended to run
docker pull microsoft/windowsservercore
docker pull microsoft/nanoserver
before you build new Windows container images to have the latest OS base image with all security fixes and bug fixes in it.
If you want to keep track which version of the Windows image you use, you can use the tags provided for each release.
Instead of using only the latest version in your Dockerfile
FROM microsoft/windowsservercore
you can append the tag
FROM microsoft/windowsservercore:10.0.14393.576
But is still recommended to update the tag after a new Windows image has been published.
You can find the tags for windowsservercore and nanoserver on the Docker Hub.
What about the framework images?
Typically you build your application on top of some kind of framework like ASP.NET, IIS or a runtime language like Node.js, Python and so on. You should have a look at the update cycles of these framework images. The maintainers have to rebuild the framework images after a new release of the Windows base image came out.
If you see some of your framework images lag behind, encourage the maintainer to update the Windows base image and to rebuild the framework image. With such updated framework images – they hopefully come with a new version tag – you can rebuild your application.
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments