This article is contributed. See the original author and article here.
Endpoint Configuration Manager (Current Branch), supports high availability configurations through various options, which include but are not limited to the following:
- Any standalone primary site can now have an additional passive mode.* site server
- Remote content library*
- SQL Server Always On
- Multiple instances of critical site system roles such as Management Points.
- Regular site DB backups
- Clients automatically remediating typical issues w/o administrative intervention
*NOTE: Feature is only available in 1806 and newer releases.
Remote Content Library
The content library is a single-instance store of content in Configuration Manager, which is used by the site to reduce the overall size of the combined body of content that is distributed distribute.
The library stores all content files for deployments (software updates, applications, and OS deployments), and copies of the library are automatically created and maintained on each site server and each distribution point.
Use the Content Library Explorer tool from the Configuration Manager tools to browse the contents of the library.
Starting in version 1806, the content library on the site server can be moved to one of the following locations:
- Another drive on the site server (use the Content Library Transfer tool)
- A separate server
- Fault-tolerant disks in a storage area network (SAN).
- A SAN is the recommended configuration due to its high availability, and it also provides elastic storage that grows or shrinks over time to meet your changing content requirements.
NOTE: This action only moves the content library on the site server. Distribution points are not impacted.
- Read and write permissions to the network path the remote content library will be located must be given to the site servers’ computer accounts.
- No components are installed on remote system.
- The site server can’t have the distribution point role because distribution points don’t support a remote content library.
- After moving the content library, the distribution point role can’t be added to the site server.
Managing the Content Library (step-by-step)
The site actually copies the content library files to the remote location, and does not delete the content library files at the original location on the site server. To free up space, an administrator must manually delete these original files.
Create a folder in a network share as the target for the content library (serversharefolder).
WARNING: Do NOT reuse an existing folder with content. For example, don’t use the same folder as your package sources because Configuration Manager removes any existing content from the location prior to copying the library.
In the Configuration Manager console”
· Switch to the Administration workspace.
· Expand Site Configuration, select the Sites node, and select the site.
On the Summary tab at the bottom of the details pane, notice a new column for the Content Library.
Click Manage Content Library on the ribbon
The Current Location field should show the local drive and path.
Enter the UNC path to the network location created in Step 1 and click OK.
Note the Status value in the Content Library column on the Summary tab of the details pane, which updates to show the site’s progress in moving the content library.
· The Move Progress (%) value displays the percentage complete.
· If there’s an error state, the status displays the error.
· Common errors include access denied or disk full.
· When complete it displays Complete.
See the distmgr.log for details.
If the content library needs to be moved back to the site server, repeat this process, but enter a local drive and path for the New Location (Example – D:SCCMContentLib), and it must be a folder name that already exists on the drive. When the original content still exists, the process quickly moves the configuration to the location local to the site server.
The following items may help troubleshoot content library issues:
- Review distmgr.log and PkgXferMgr.log on the site server, and the smsdpprov.log on the distribution point for possible indicators of the failures.
- The Content Library Explorer tool.
- Check for file locks by other processes (antivirus software, etc.).
- Exclude the content library on all drives from automatic antivirus scans, as well as the temporary staging directory, SMS_DP$, on each drive.
- Validate the package from the Configuration Manager console in case of hash mismatches.
- As a last option, redistribute the content, which should resolve most issues.
Site Server High Availability
Introduced in the 1806 release of Configuration Manager, high availability for the site server role is achievable by installing a second passive mode site server. This passive site server is in addition to the existing (active mode) site server, and it remains available for immediate use, when needed.
While in passive mode, a site server:
- Uses the same site DB as the active server.
- Only reads from the site database.
- It will begin writing to the database if promoted to active.
- Uses the same content library as the active mode server.
Manually promoting the passive server will make it the active server; thereby, switching the active server to passive mode. Since the site server role is the only role that is switched, the other site system roles on the original active mode server remain available so long as that computer is accessible.
NOTE: This feature is not enabled by default.
- Passive mode site servers can be on-premises or cloud-based in Azure.
- Both site servers must be joined to the same Active Directory domain.
- The site is a standalone primary site.
- Both site servers must use the same site database, which must be remote to each site server.
- Both site servers need sysadmin permissions on the SQL Server instance for the DB.
- The SQL Server that hosts the site database can use a default instance, named instance, SQL Server cluster, or a SQL Server Always On availability group.
- The content library must be on a remote network share, and both site servers need Full Control permissions to the share and its contents.
- The site server can’t have the distribution point role.
- The site server in passive mode:
- Must meet all prerequisites for installing a primary site.
- Must have its computer account in the local admins group on the active site server.
- Installs using source files that match the version of the site server in active mode.
- Can’t have a site system role from any site prior to installing the site server in passive mode role.
- Both site servers can run different OS versions, as long as both are supported by Configuration Manager.
- Only one passive mode server at each primary site.
- Passive mode servers are not supported in a hierarchy (CAS with a child primary site(s)).
- It must be a standalone primary site.
- Secondary sites do not support passive mode site servers.
- Promotion of the passive mode server to active mode is manual.
- No automatic failover.
- Roles can’t be installed on the new server until it is added as a passive site server.
- NOTE: After the passive mode server is installed, additional roles can be installed as necessary (SMS Provider, management point, etc.).
- Databases for roles like the reporting point, should be hosted on a server that’s remote from both site servers.
- The passive mode server does NOT receive the SMS Provider. Since a connection to a provider is required to manually promote the passive site server, install at least one additional instance of the SMS Provider on server that is NOT the active site server.
- The Configuration Manager console doesn’t automatically install on the passive server.
Add Passive Mode Server (step-by-step)
In the Configuration Manager console:
· Click the Administration workspace
· Expand Site Configuration
· Select the Sites node
· Click Create Site System Server in the ribbon.
On the General page of the Create Site System Server Wizard:
· Specify the server to host the site server in passive mode.
NOTE: The server you specify can’t host any site system roles before installing a site server in passive mode.
On the System Role Selection page, select only Site server in passive mode.
The following prerequisite checks are performed at this stage:
· Selected server isn’t a secondary site server
· Selected server isn’t already a site server in passive mode
· Content library is in a remote location
If these initial prerequisite checks fails, you can’t continue past this page of the wizard.
On the Site Server In Passive Mode page, provide the following information:
· Choose one of the following options:
· Copy installation source files over the network from the site server in active mode:
· This option creates a compressed package and sends it to the new site server.
· Use the source files at the following location on the site server in passive mode:
· This is a local path that already contains a copy of the source files. Make sure this content is the same version as the site server in active mode.
· (Recommended) Use the source files at the following network location:
· Specify the path directly to the contents of the CD.Latest folder from the site server in active mode. (Example: ServerSMS_ABCCD.Latest)
· Specify the local path to install Configuration Manager on the new site server.(C:Program FilesConfiguration Manager)
Complete the wizard.
Navigate to the Monitoring workspace, and select the Site Server Status node in the ConfigMgr console for detailed installation status. The state for the passive site server should display as Installing.
· For more detail, select the server and click Show Status. This action opens the Site Server Installation Status window, and the state will OK for both servers when the process completes.
For more information on the setup process, see the flowchart in the appendix
NOTE: All Configuration Manager site server components are in standby on the site server in passive mode.
Site server promotion
As with any backup/recovery plan, practice the process to change site servers, and consider the following:
- Practice a planned promotion, where both site servers are online as well as an unplanned failover, by forcibly disconnecting or shutting down the site server in active mode.
- Before a planned promotion:
- Check the overall status of the site and site components, making sure everything is healthy.
- Check content status for any packages actively replicating between sites.
- Do not start any new content distribution jobs.
NOTE: If file replication between sites is in progress during failover, the new site server may not receive the replicated file. If this happens, redistribute the software content after the new site server is active.
Process to promote the site server in passive mode to active mode
In order to access the site and promote a server from passive to active mode, access to an instance of the SMS Provider is absolutely necessary.
IMPORTANT: By default, only the original site server has the SMS Provider role. If this server is offline, no provider is available, and access to the site is not possible. When a passive site server is added, the SMS Provider is NOT automatically installed, so ensure that at least one additional SMS Provider role is added to the site for a highly available service.
If the console is unable to connect to the site because the current site server (SMS Provider) is offline, specify the other site server (additional SMS Provider) in the Site Connection window.
In the Configuration Manager console:
· Navigate to the Administration workspace
· Expand Site Configuration
· Select the Sites node
· Select the site, and then switch to the Nodes tab.
· Select the site server in passive mode, and then click Promote to active in the ribbon.
· Click Yes to confirm and continue.
Refresh the console node, and the Status column for the server being promoted should display in the Nodes tab as Promoting.
After the promotion is complete, the Status column should show OK for both the new (active) site server, and the new passive site server.
· The Server Name column for the site should now display the name of the new site server in active mode.
For detailed status, navigate to the Monitoring workspace, and select the Site Server Status node. The Mode column will identify which server is active/passive.
When promoting a server from passive to active mode, select the site server to be promoted, and then choose Show Status from the ribbon, which opens the Site Server Promotion Status window for additional detail.
If the active site server goes offline, the following process begins:
- The passive site server will try to contact the active site server for 30 minutes.
- If the offline server comes back before this 30 minute timeframe, it’s successfully notified, and the change proceeds gracefully.
- Otherwise the passive site server forcibly updates the site configuration for it to become active.
- If the offline server comes back after this time, it first checks the current state in the site database, and then proceeds with demoting itself to the passive site server.
NOTE: During this 30 minute waiting period, there is no active site server. However, clients will still communicate with client-facing roles such as management points, software update points, and distribution points.
Users can also continue to install software that’s already deployed, but site administration is NOT possible in this time period.
If the offline server is damaged such that it can’t return, delete this site server from the console. Then create a new site server in passive mode to restore a highly available service.
Passive site servers should be monitored daily to ensure its Status remains OK and ready for use. This is done in the Monitoring workspace of the ConfigMgr console by selecting the Site Server Status node.
- View both site servers and their current status.
- Also view status in the Administration workspace.
- Expand Site Configuration, and select the Sites node. Select the site, and then switch to the Nodes tab.
SQL Server Always On
SQL Server Always On provides a high availability and disaster recovery solution for the site database.
An Always On availability group supports a replicated environment for a discrete set of user databases, known as availability databases. When created for high availability the availability group’s of databases fail over together.
- An availability group supports one set of primary databases and one to eight sets of corresponding secondary databases.
- Secondary databases are not backups, so continue to back up your databases and their transaction logs on a regular basis.
Each set of availability databases is hosted by an availability replica. Two types of availability replicas exist:
- Single primary replica that hosts the primary databases
- One to eight secondary replicas, which hosts a set of secondary databases and serve as a potential failover targets for the availability group.
The primary replica makes the primary databases available for read-write connections from clients, and sends transaction log records of each primary database to every secondary database.
- Known as data synchronization and occurs at the database level.
- Every secondary replica caches the transaction log records (hardens the log) and then applies them to its corresponding secondary database.
- Data synchronization occurs between the primary database and each connected secondary database, independently of the other databases.
Supported Scenarios for Configuration Manager
The following situations are supported when using SQL Server Always On availability groups in Configuration Manager.
- Creating an availability group
- Configuring a site to use an availability group
- Add or remove synchronous replica members from an availability group hosting the site database
- Configuring asynchronous commit replicas
- Recovering a site from an asynchronous commit replica
- Moving a site database out of an availability group to a standalone (default or named instance) of SQL Server
NOTE: The step-by-step details for performing each of these scenarios are described in the following documentation:
Configure SQL Server Always On availability groups for Configuration Manager
Prerequisites for all Scenarios
Configuration Manager accounts and permissions
Site server to replica member access
The site server’s computer account must be a member of the Local Administrators group on each member of the availability group.
Each replica in the availability group must run a version of SQL Server that’s supported by the environment’s version of Configuration Manager.
- When supported by SQL Server, different nodes of an availability group can run different versions of SQL Server.
NOTE: SQL Server Enteprise Edition must be used
A domain user (service) account a non-domain account can be used and each replica in a group can have a different configuration.
- Use lowest possible permissions.
- Certificates must be used for non-domain accounts.
Availability group configurations
- Group must have one primary replica.
- Use the same number/type of replicas that’s supported by the version of SQL Server in use.
- An asynchronous commit replica can be used to recover a synchronous replica.
WARNING: Configuration Manager doesn’t support failover to use the asynchronous commit replica as the site database because Configuration Manager doesn’t validate the state of the asynchronous commit replica to confirm it’s current. This can put the integrity of the site and data at risk because (by design) such a replica can be out of sync.
Each replica member must have the following configuration:
- Use default or a named instance
- The Connections in Primary Role setting is Yes
- The Readable Secondary setting is Yes
- Enabled for Manual Failover
NOTE: Configuration Manager supports using the availability group synchronous replicas when set to Automatic Failover. Set Manual Failover when:
- Configuration Manager Setup specifies the use of the site database in the availability group.
- Any update to Configuration Manager is installed. (Not just updates that apply to the site database).
Replica member location
All replicas in an availability group must be either on-premises, or all hosted on Microsoft Azure.
- Combinations within a group that include both on-premises and Azure isn’t supported.
Configuration Manager Setup must be able to connect to each replica, so ensure the following ports are open:
- RCP Endpoint Mapper: TCP 135
- SQL Server Service Broker: TCP 4022
- Must stay open following setup
- SQL over TCP: TCP 1433
- Must stay open following setup
NOTE: Custom ports for these configurations are supported so long as they are the same custom ports on the endpoint and on all replicas in the availability group.
The must be at least one listener in an availability group.
- Configuration Manager uses the virtual name of the listener when configured to use the site database in the availability group.
- Although an availability group can contain multiple listeners.
Each secondary replica in the availability group must have a SQL Server file path that’s identical to the file path for the site database files on the current primary replica. Otherwise, setup will fail to add the instance for the group as the new location of the site DB.
NOTE: The local SQL Server service account must have Full Control permission to this folder.
This path is only required when ConfigMgr setup is used to specify the database instance in the availability group. Upon completion, the path can be deleted from the secondary replica servers.
- Primary replica server is a new installation of SQL Server 2014, which (by default) stores the .MDF and .LDF files in C:Program FilesMicrosoft SQL ServerMSSQL12.MSSQLSERVERMSSQLDATA.
- Secondary replica servers are upgraded to SQL Server 2014 from previous versions, but the servers keep the original file path as: C:Program FilesMicrosoft SQL ServerMSSQL10.MSSQLSERVERMSSQLDATA.
- Before the site database can be moved to this availability group, each secondary replica server must have the path used by the primary, even if the secondaries won’t use the location
- As a result C:Program FilesMicrosoft SQL ServerMSSQL12.MSSQLSERVERMSSQLDATA must be created on each secondary.
- Grant the SQL Server service account on each secondary replica full control access to the newly created location, and then run Setup to move the site database.
Configure the database on a new replica
The database of each replica should be configured with the following settings:
- Enable CLR Integration.
- Set Max text repl size to 2147483647
- Set the database owner to the SA account
- Turn ON the TRUSTWORTY setting.
- Enable the Service Broker
NOTE: Only make these configurations on a primary replica. The primary must be failed-over to a secondary replica in order to configure that secondary, which also makes the secondary the new primary replica.
Run the following SQL script to verify database configurations for both primary and secondary replicas. Before you can fix an issue on a secondary replica, change that secondary replica to be the primary replica.
SET NOCOUNT ON DECLARE @dbname NVARCHAR(128) SELECT @dbname = sd.name FROM sys.sysdatabases sd WHERE sd.dbid = DB_ID() IF (@dbname = N'master' OR @dbname = N'model' OR @dbname = N'msdb' OR @dbname = N'tempdb' OR @dbname = N'distribution' ) BEGIN RAISERROR(N'ERROR: Script is targetting a system database. It should be targeting the DB you created instead.', 0, 1) GOTO Branch_Exit; END ELSE PRINT N'INFO: Targetted database is ' + @dbname + N'.' PRINT N'INFO: Running verifications....' IF NOT EXISTS (SELECT * FROM sys.configurations c WHERE c.name = 'clr enabled' AND c.value_in_use = 1) PRINT N'ERROR: CLR is not enabled!' ELSE PRINT N'PASS: CLR is enabled.' DECLARE @repltable TABLE ( name nvarchar(max), minimum int, maximum int, config_value int, run_value int ) INSERT INTO @repltable EXEC sp_configure 'max text repl size (B)' IF NOT EXISTS(SELECT * from @repltable where config_value = 2147483647 and run_value = 2147483647 ) PRINT N'ERROR: Max text repl size is not correct!' ELSE PRINT N'PASS: Max text repl size is correct.' IF NOT EXISTS (SELECT db.owner_sid FROM sys.databases db WHERE db.database_id = DB_ID() AND db.owner_sid = 0x01) PRINT N'ERROR: Database owner is not sa account!' ELSE PRINT N'PASS: Database owner is sa account.' IF NOT EXISTS( SELECT * FROM sys.databases db WHERE db.database_id = DB_ID() AND db.is_trustworthy_on = 1 ) PRINT N'ERROR: Trustworthy bit is not on!' ELSE PRINT N'PASS: Trustworthy bit is on.' IF NOT EXISTS( SELECT * FROM sys.databases db WHERE db.database_id = DB_ID() AND db.is_broker_enabled = 1 ) PRINT N'ERROR: Service broker is not enabled!' ELSE PRINT N'PASS: Service broker is enabled.' IF NOT EXISTS( SELECT * FROM sys.databases db WHERE db.database_id = DB_ID() AND db.is_honor_broker_priority_on = 1 ) PRINT N'ERROR: Service broker priority is not set!' ELSE PRINT N'PASS: Service broker priority is set.' PRINT N'Done!' Branch_Exit:
Unsupported SQL Server Configurations
- Basic availability groups: (SQL Server 2016 Standard). These groups don’t support read access to secondary replicas that is required by ConfigMgr
- Failover cluster instances (pre 1810 releases)
- 1810 and newer releases allows for a highly available site with fewer servers by using a SQL cluster and a site server in passive mode
- MultiSubnetFailover (pre 1906 releases)
- Starting with the 1906 release of ConfigMgr, MultiSubnetFailover is now supported for HA configurations.
SQL Servers Hosting Additional Availability Groups
SQL Servers hosting more than one availability group require specific settings at the time of Configuration Manager Setup and when installing updates for Configuration Manager. Each replica in each availability group must have the following configurations:
- Manual Failover
- Allow any read-only connection
Unsupported Database Usage
- Configuration Manager supports only the site database in an availability group. The Reporting and WSUS databases used by Configuration Manager are NOT supported in a SQL Server Always On availability group:
- Pre-existing database: When configuring an availability group, restore a copy of an existing Configuration Manager database to the primary replica.
When moving a site database to an availability group, Setup tries to process database roles on the secondary replicas, and the ConfigMgrSetup.log file shows the following error, which is safe to ignore:
ERROR: SQL Server error: [Microsoft][SQL Server Native Client 11.0][SQL Server]Failed to update database “CM_AAA” because the database is read-only. Configuration Manager Setup 1/21/2016 4:54:59 PM 7344 (0x1CB0)
To expand a standalone primary site to include a CAS, the primary site’s database must first be temporarily removed from the availability group.
Changes for site backup
Run the built-in Backup Site server maintenance task to back up common Configuration Manager settings and files.
- Do NOT use the .MDF or .LDF files created by that backup.
- Make direct backups of these database files by using SQL Server.
- Set the recovery model of the site database to Full, which is required for Configuration Manager use in an availability group.
- Monitor and maintain the size of the transaction log because the transactions aren’t hardened until a full backup of the database or transaction log occur.
Changes for site recovery
If at least one node of the availability group is still functional, use the site recovery option to Skip database recovery (Use this option if the site database was unaffected).
If all nodes of an availability group are lost, the group must be recreated before site recover is attempted.
- Recreate the group
- Restore the backup
- Reconfigure SQL.
- Use the site recovery option to Skip database recovery (Use this option if the site database was unaffected).
Changes for reporting
Install the reporting service point
The RSP doesn’t support using the listener virtual name of the availability group, or hosting its database in a SQL Server Always On availability group.
- RSP installation sets the Site database server name to the virtual name that’s specified as the listener.
- Change this setting to a computer name and instance of a replica in the availability group.
- Installing addition RSPs on each replica node increases availability when a replica node is offline
- Each RSP should be configured to use its own computer name.
- The active RSP can be changed in the ConfigMgr console (Monitoring > Reporting > Reports > Report Options)
Content Library Explorer
Content Library Explorer be used for the following:
- Explore the content library on a specific distribution point
- Troubleshoot issues with the content library
- Copy packages, contents, folders, and files out of the content library
- Redistribute packages to the distribution point
- Validate packages on remote distribution points
- Run the tool using an account that has administrative access to:
- The target distribution point
- The WMI provider on the site server
- The Configuration Manager (SMS) provider
- Only the Full Administrator and Read-Only Analyst roles have sufficient rights to view the tool’s info.
- Other roles can view partial information.
- The Read-Only Analyst can’t redistribute packages from this tool.
- Run the tool from any computer, as long as it can connect to:
- The target distribution point
- The primary site server
- The Configuration Manager provider
- If the distribution point is co-located with the site server, administrative access to the site server is still required.
When you start the tool (ContentLibraryExplorer.exe),
- Enter the FQDN of the target distribution point.
- If the distribution point is part of a secondary site, it prompts for the FQDN of the primary site server, and the primary site code.
- In the left pane, view the packages that are distributed to this distribution point, and explore their folder structure. This structure matches the folder structure from which the package was created.
- When a folder is selected, its files are displayed in the right pane. This view includes the following information:
- File name
- File size
- Which drive it’s on
- Other packages that use the same file on the drive
- When the file was last changed on the distribution point
The tool also connects to the Configuration Manager provider to determine which packages are distributed to the distribution point, and whether they’re actually in the distribution point’s content library or in a “pending” status. Pending packages will not have any available actions.
Some packages are present on the distribution point, but not visible in the Configuration Manager console, and are marked with an asterisk (*). Most (if not all) actions will not be available for these packages.
There are three primary reasons for disabled packages:
- The package is the ConfigMgr client upgrade.
- The user account can’t access the package, likely due to role-based administration.
- The package is orphaned on the distribution point.
Validate packages by using Package > Validate on the toolbar.
- First select a package node in the left pane (don’t select a content or a folder).
- The tool connects to the WMI provider on the distribution point for this action.
- When the tool starts, packages that are missing one or more contents are marked invalid. Validating the package reveals which content is missing. If all content is present but the data is corrupted, validation detects the corruption.
Redistribute packages using Package > Redistribute on the toolbar.
- First select a package node in the left pane.
- This action requires permissions to redistribute packages.
Use Edit > Copy to copy packages, contents, folders, and files out of the content library to a specified folder.
- You can’t copy the content library itself. Select more than one file, but you can’t select multiple folders.
Search for packages using Edit > Find Package.
- This use the query to search in the package name and package ID.
- Cannot be used to manipulate the content library directly, which may result in malfunctions.
- Can redistribute packages, but only to the target distribution point.
- When the distribution point is co-located with the site server, package data can NOT be validated. Use the Configuration Manager console instead.
- The tool still inspects the package to make sure that all the content is present, though not necessarily intact.
- Content can NOT be deleted.
Passive Site Server Setup
Promote Site Server (planned)
Promote Site Server (unplanned)
- High Availability Options for Configuration Manager
- The Content Library in Configuration Manager
- Site Server High Availability
- Content Library Explorer
- Prerequisites for Installing a Primary Site
- Supported OS Configurations for Configuration Manager
- Plan for the SMS Provider
- Configuration Manager Tools
- Overview of Always On Availability Groups
- SQL Server Always On for Configuration Manager
- Supported SQL Server Versions for Configuration Manager
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.