SCOM 2016 – Upgrade Notes from the Field

Upgrading from SCOM 2012 R2 to SCOM 2016 is theoretically no such big deal. BUT sometimes you could face issues at the customer’s infrastructure, which force you to take some extra hurdle. This post should give you a high level overview of different migration scenarios and additionally some pitfalls you could meet upgrading to SCOM 2016.

High level upgrade path

There are 3 ways to upgrade a SCOM 2012 R2 environment.

1. Side-by-side migration (“Slow Motion”)

• This is probably the way which has almost no risks, but takes a long time to finish and has a consequence that you loose old data. Why is this? You install a brand new SCOM 2016 management group, having brand new databases (OperationsManager / OperationsManagerDW / OperationsManagerAC). If needed you also install separate Web Console, Reporting and if needed the ACS role also on a dedicated (management) server. I think the best option is to install all these SCOM 2016 roles on a brand new Windows Server 2016 server and the databases on SQL Server 2016. This way you have the latest and greatest technologies available and you are armed for the next couple of years. Having this in place you are able to dual-home (multi-homing) the agent which is sending data to both management groups SCOM 2012 R2 and SCOM 2016. There is a good article on TechNet Wiki how to configure multi-homing if you have multiple AD forests or here if you have agents deployed in the same AD forest. As soon you have the new management group up and running you need to migrate all management packs, channels, subscriptions, overrides, roles etc. There are ways to export and import this stuff, but I recommend if you are choosing this upgrade path, then I would start configuring SCOM from scratch. Especially creating new overrides and documenting them will give you a chance to have a well configured and documented SCOM environment. One huge advantage of this upgrade path is, that you are able to upgrade to new versions of existing management packs, implementing new management packs and testing them thoroughly with no impact on your production SCOM environment until you switch management group and turn on notifications. This approach has also few disadvantages:
• It takes usually a long time to finish this migration.
• There are 2 management groups to maintain.
• The amount of work to tune the management packs should not be underestimated.
• Dual-homing an agent could lead to some more stress on the agent server.

2. SCOM In-place only upgrade (“Big Bang”)

• If you decide to go for an in-place upgrade you are taking a much faster but also “risky” path, which needs more pre-work, testing and in case of failures also some plans to revert the changes using backups and/or VM snapshots. An in-place upgrade is in theory not that much of a problem and also fully supported by Microsoft. The first step is to run the SCOM 2016 setup on a management server which will discover the roles on the management server and upgrade the server itself and also the SCOM databases to SCOM 2016. If you managed to successfully upgrade the first management server / management group, then you go for the next management server, ACS Collector, Gateways, Console, Web Console and Reporting Server. As soon you have upgraded all components you are all done. Sounds easy, but believe me, there are plenty of things that could fail. This approach has also few disadvantages:
• Because you upgrade SCOM only, the operating system stays the same. Of course you could theoretically in-place upgrade the operating system as well, but I really don’t encourage you to do so. If you need to upgrade SCOM and the operating system as well, please check the next upgrade option.
• All your SCOM configurations bad or good will stay. If your management group is badly configured it will stay badly configured – an upgrade won’t change anything.
• You need to check if the management packs work with SCOM 2016, especially third party or community MP’s. Please ask the vendor BEFORE you start the upgrade.
• Make sure you meet the system requirements for SCOM 2016 .
• Remember there are also 3rd party connectors in SCOM which might are not supported by SCOM 2016.

3. SCOM In-place upgrade and OS upgrade (“Big Bang++”)

• If you decide to go for an in-place upgrade and you also want to upgrade the operating system to Windows Server 2016 in your environment, then this is an elegant way to achieve this goal. The risks are the same as “in-place only” upgrade but in addition you need to have a good plan how to switch the SCOM agents and ACS Forwarders to the new management servers. Before you start upgrading, make sure you have new Windows Server 2016 servers installed, which will become the new management servers. Step 1 is to run in-place upgrade on an “old” SCOM 2012 R2 management server (make sure it meets SCOM 2016 system requirements). If this is finished upgrade the other SCOM 2012 R2 management servers to SCOM 2016 and also ACS Collector, Gateways, Console, Web Console and Reporting Server. Step 2 if your management group is upgraded successfully install SCOM 2016 management servers on the fresh installed Windows Server 2016 servers. Depending on your SCOM environment, but if you have ACS installed, you could also install ACS Collector on a additional dedicated SCOM 2016 management server running on Windows Server 2016. Step 3 move the Windows / Linux agents, ACS Forwarders to the new management servers / ACS Collector. Step 4 uninstall the “old” management servers from the management group. If you have Web Console and/or Reporting installed you could simply uninstall the features from the “old” SCOM servers and reinstall it on new Windows Server 2016 server pointing to the SCOM 2016 deployment. I recommend uninstalling Reporting and Web Console BEFORE you upgrade the management group. This scenario has the same problems as an “in-place only” scenario but additionally you have to be aware of few more things:
• Switching the Windows / Linux agents or ACS Forwarders to the new management servers could take some time and depending on the amount of “clients” Step 3 needs to be planned carefully.
• If you don’t have the agents controlled by the SCOM Console you need to prepare some PowerShell scripts for moving the agents / ACS Forwarders to the new management servers.
• Remember to install certificates for Linux  or Windows agents monitoring on the new management servers.
• Remember to set the SPNs for the new management servers.
• If you changed settings (Registry)  on your old management servers, check if you need to make these settings on your new management servers as well.

Some important things to keep in mind

• IMPORTANT: Test your migration scenario in a lab! I REALLY REALLY REALLY urge you to do so, if you don’t have a lab build one NOW!
• IMPORTANT: Make VM Snapshots of your entire management group BEFORE you start and DURING migration where it makes sense! This will save your a**!
• Apply SCOM 2016 UR3 immediately after upgrade.
• Depending on your scenario make sure you meet always the system requirements of SCOM 2012 R2 and SCOM 2016. This is very important if you are doing an in-place upgrade so all “old” components like SQL Server and operating system meet the SCOM 2016 requirements AND SCOM 2012 R2 requirements.
• Make sure you upgrade SCOM 2012 R2 to the latest UR before you upgrade.
• SCOM 2012 R2 agent (try to keep the latest UR!) is forward compatible to communicate to SCOM 2016 MG, but also SCOM 2016 agent is backwards compatible with SCOM 2012 R2 MG.
• Be aware that there are rewritten management packs available which you might want to consider to install in your SCOM 2016 environment, in case you have older versions in your SCOM 2012 R2 MG like Windows Server Active Directory (old: 6.0.8321.0 | new: 10.0.2.1) or Windows Server DNS (old: 6.0.7000.0 | new: 10.0.6.0 note: there is a known bug ) . If you install new MP’s that differ from your old environment, you need to check your overrides carefully! This article from MVP Cameron Fuller might help to understand the basic principle of overrides migration.
• Make sure your third-party management packs work with SCOM 2016, having a side-by-side migration let’s you easily test and deploy new management packs without interrupting the production SCOM 2012 R2 environment.
• Make sure you adapt your antivirus exclusion rules to SCOM 2016.
• If SCOM 2016 agent is grey on domain controllers check Kevin Holman’s post.
• Set the new SCOM 2016 registry tweeks after upgrade.
• If you click on the scheduling maintenance mode view in SCOM 2016 it will shoot an error. Apply this fix here.
• SCOM 2016 UR3 agent does still have an issue that it crashes the IIS Application Pool running .NET 2.0 runtime . I would recommend using SCOM 2012 R2 agent until this problem is fixed. SCOM 2012 R2 agent is forward compatible with SCOM 2016 management group.
• If your upgrade setup fails with an error in OpsMgrSetupWiz.Log that there are already some components installed…

MSI (s) (BC:F8) [14:18:59:579]: Product: Microsoft Monitoring Agent — The Microsoft Monitoring Agent cannot be installed on a computer on which the Operations Manager management server, gateway server, Operations console, operational database, web console, System Center Essentials, or System Center Service Manager is already installed. The Microsoft Monitoring Agent cannot be installed on a computer on which the Operations Manager management server, gateway server, Operations console, operational database, web console, System Center Essentials, or System Center Service Manager is already installed.
Action ended 14:18:59: _AbortCoreComponentPresent. Return value 3.

…check the installed products on your management server run wmic /output:C:\temp\InstallList.txt product get analyze the list for unwanted software. Uninstall them! If you cannot uninstall them, but they appear on the list check the following registry keys and if necessary backup and delete them

• HKEY_Classes_Root\Installer\Products
• HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall
• HKU\USER-SID-HERE\Software\Microsoft\Installer\Products
• HKLM\Software\Classes\Installer\Products
• HKLM\Software\Wow6432Node\Microsoft\Windows\
  CurrentVersion\Uninstall

• If your upgrade fails with an error in OpsMgrSetupWiz.Log that there is a missing assemply like this…

Error: :PopulateUserRoles: failed : Threw Exception.Type: System.TypeLoadException, Exception Error Code: 0x80131522, Exception.Message: Could not load type ‘SCTrace’ from assembly ‘Microsoft.EnterpriseManagement.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’. [10:26:08]: Error: :StackTrace:   at Microsoft.Mom.Sdk.UserRoleSetup.SetupProgram. populateUserRoles(String adminRoleGroup, String sdkAccount, InstallTypes installType, String installDirectory, Boolean overwriteExistingUsers) at Microsoft.EnterpriseManagement.OperationsManager.Setup. ServerConfiguration.PopulateUserRoles(String adminRoleGroup, String sdkAccount, String installDirPath)

…follow these steps

1. Delete the key HKEY_CLASSES_ROOT\Installer\UpgradeCodes\ C96403E8AD6025B4F9E1FE9C574E34AE
2. Enable developer mode in SetupChainerUI.exe.config

<configuration>
  <runtime>
    <developmentMode developerInstallation=”true”/>
  </runtime>
< /configuration>

3. Create a DEVPATH environment variable with the value DEVPATH=C:\Users\[User]\AppData\Local\SCOM\Setup\
4. Re-run the setup

I encountered this error during my past upgrade. Thanks to Tim Helton (MSFT) for providing this solution

• Remove software from your management server you want to upgrade:
• SCOM 2007 R2 Authoring console
• Sliect MP Author

I experienced issues / failures if those software was installed and I ran the setup.

The high level migration scenarios are not meant to be a step-by-step guide, instead it should provide an overview how you could migrate to SCOM 2016. In addition this post is not a complete list of issues you could run into, but it should help you to avoid or troubleshoot certain issues.