SCOM – How Data is Encrypted

Recently I got a question from a customer how SCOM traffic is encrypted. Well, I knew that the traffic IS encrypted, but how the encryption works, that is a different story.

First we need to know, about what traffic we are talking about. Is it the communication between agents , respectively healthservices? Is it the encryption of RunAs accounts / credentials within the communication channel? Or, are we talking about the encryption of RunAs accounts within the SCOM database? On TechNet you will find an article talking about the communication and encryption https://technet.microsoft.com/en-us/library/bb735408.aspx but what is the context having certificates or Kerberos in place. To get the full picture, we need to answer these questions.

No one else could answer these questions better than Microsoft itself and therefore of course “Mr. SCOM” Kevin Holmann. All credits to him, he provided me with this very interesting information and letting me publishing it. Thank you Kevin!

Let’s first talk about the healthservice to healthservice communication.

1, Healthservice to Healthservice Encryption and Authentication:

Communication among these Operations Manager components begins with mutual authentication. If certificates are present on both ends of the communications channel (and enabled for use in the registry for the healthservice), then certificates will be used for mutual authentication.  Otherwise, the Kerberos version 5 protocol is used. If any two components are separated across an untrusted domain/forest boundary that doesn’t support Kerberos, then mutual authentication must be performed using certificates.

If Kerberos is available, the agent is authenticated via Kerberos, and then still using Kerberos, the data channel is encrypted using Kerberos AES or RC4 cypher.  A by-product of the Kerberos authentication protocol is the exchange of the session key between the client and the server. The session key may subsequently be used by the application to protect the integrity and privacy of communications. The Kerberos system defines two message types, the safe message and the private message to encapsulate data that must be protected, but the application is free to use a method better suited to the particular data that is transmitted.

If certificates are used for mutual authentication, the same certificates are used to encrypt the data in the channel.

=> Agents are initially authenticated via Kerberos, or Certificates.  Then that same protocol is used for encryption of the channel – per:

https://technet.microsoft.com/en-us/library/bb735408.aspx

From the agent to the gateway server, the Kerberos security package is used to encrypt the data, because the gateway server and the agent are in the same domain. The alert is decrypted by the gateway server and re-encrypted using certificates for the management server. After the management server receives the alert, the management server decrypts the message, re-encrypts it using the Kerberos protocol, and sends it to the RMS where the RMS decrypts the alert.

2. RunAs Credential Encryption Decryption in the Channel

In addition to the default channel authentication and encryption, there is an additional layer of encryption for RunAs account credentials.  There is a self-signed cert that gets generated under the OperationsManager folder in certificates, which gets generated when the healthservice starts, and updated/replaced as needed.  This certificate is solely used to protect RunAs account credentials in the transmission from the management server to the agent.  It doesn’t have any impact on how they are stored on the agent itself, and is not used for authentication.  On the agent RunAs accounts are stored in the registry and protected using DPAPI.  RunAs accounts are sent to the agent as part of the OpsMgrConnector.Config.xml file.  In that file the RunAs accounts are encrypted, base64 encoded and placed in the Message/State/SecureData element.  The encryption key is the agent self-signed certificate.  When the agent starts up it creates or gets the existing certificate and publishes the public key to its management server, which then submits it to the database via the SDK service.  When the configuration service generates configuration for an agent, it looks up the public key for that agent, and then uses that key to encrypt the SecureData part of the configuration XML.  The agent has a cert lifetime set of 1 year and will generate and transmit a new certificate when it is getting near expiration.

You will see associated events for this cert:

Log Name:      Operations Manager
Source:        HealthService
Date:          12/3/2015 8:23:12 AM
Event ID:      7006
Task Category: Health Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     {ServerName]
Description:
The Health Service has published the public key [4A A8 71 8B 0D 3F 9E 9D 4A 59 44 D8 EE BC B1 42 ] used to send it secure messages to management group [MG Name].  
This message only indicates that the key is scheduled for delivery, not that delivery has been confirmed.

3. Encryption for RunAs Accounts Stored in the SCOM DB

There IS an RMS Encryption key.  Well, there “was”.  However – it isn’t really an “RMS” key anymore, it is simply a “RunAs Account password encryption key”.  The key is used to encrypt the passwords of the RunAs account credentials in the database, and then decrypt them for use. The first management server generates it, and all subsequent management servers get their copy key when they are installed. We no longer care anymore, because the management servers are federated for the SDK and config, and as long as you have one MS left in a DR scenario – you simply add management servers and the key is preserved.  If you lose ALL your management servers, you install using the /Recover switch, and a name of a previously existing MS – and this automatically regenerates a new key, and this is why you must re-enter your RunAs account passwords for all of them in this scenario, to re-encrypt them in the database using the new key.

This works very similarly to how it worked in SCOM 2007, which in SP1 Microsoft released the CREATE_NEWKEY=1 setup option.  However, it happens automatically now – and is a hands off process – so we (Microsoft) don’t discuss it or back it up, since it isn’t necessary. The key is stored in the Management Server registry, to my understanding. 

• SCOM 2007:  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\MOMBins
• SCOM 2012 (and SCSM):  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\System Center\2010\Common\MOMBins

The encryption is done via Windows CryptoAPI services (Crypt32.dll) using CryptProtectData: https://msdn.microsoft.com/en-us/library/windows/desktop/aa380261(v=vs.85).aspx . It uses the pOptionalEntropy parameter using a random generator function to add additional entropy to the encryption. From the article liked, here is the relevant detail about the encryption strength:

DPAPI Security

DPAPI provides an essential data protection capability that ensures the confidentiality of protected data while allowing recovery of the underlying data in the event of lost or changed passwords. The password-based protection provided by DPAPI is excellent for a number of reasons.

• It uses proven cryptographic routines, such as the strong Triple-DES algorithm in CBC mode, the strong SHA-1 algorithm, and the PBKDF2 password-based key derivation routine.
• It uses proven cryptographic constructs to protect data. All critical data is cryptographically integrity protected, and secret data is wrapped by using standard methods.
• It uses large secret sizes to greatly reduce the possibility of brute-force attacks to compromise the secrets.
• It uses PBKDF2 with 4000 iterations to increase the work factor of an adversary trying to compromise the password.
• It sanity checks MasterKey expiration dates.
• It protects all required network communication with Domain Controllers by using mutually authenticated and privacy protected RPC channels.
• It minimizes the risk of exposing any secrets, by never writing them to disk and minimizing their exposure in swappable RAM.
• It requires Administrator privileges to make any modifications to the DPAPI parameters in the registry.
• It uses Windows File Protection to help protect all critical DLLs from online changes even by processes with Administrator privileges.

SSLv3 is not required by SCOM. When SCOM uses authentication, and SSLv3 is not present, the standard OS security handshake will negotiate another method, such as TLS.

I hope this answers your questions and provides you with solid information.