Azure Automation – Up / Down Scale Azure VM

car

Recently I made a couple of demos how you could benefit from Azure Automation acting on certain alerts. One demo was, if an OMS or Azure Monitor alert is triggered because of high CPU on your Azure VM, a webhook will be called and scale up the Azure VM to a predefined size.

webhook-alert

In Azure Monitor you are able to create an Alert and set the threshold to e.g. 80%…

image

…and then call a webhook…

image

Note: Azure Monitor has already built-in alert action to start / stop a virtual machine or even scale up / down. These actions install and call pre-configured Azure Automation runbooks from Microsoft. But this post should show you in a transparent way how you can achieve it, doing it yourself.

As you probably know in OMS you are also able to provide a webhook in your alert settings…

image

Which ever way you choose Azure Monitor or OMS, it does not matter. I just want to provide you the PowerShell script I used in Azure Automation. Although it is far from complete and production level, but as demo it serves it’s purpose Smile.

# Get automation credential for authenticating and resizing the VM
$Credential = Get-AutomationPSCredential -Name 'AzureScaleUser'
# Size of the VM like Standard_D1_v2, Standard_D2_v2
$HWProfile = Get-AutomationVariable -Name 'ScaleUpSize'
# Resource group where the VM is living in
$ResourceGroup = Get-AutomationVariable -Name 'ResourceGroup'
# Subscription which hosts the VM / account etc.
$SubscriptionId = Get-AutomationVariable -Name 'SubscriptionID'
# VM name you want to up / down scale
$VMName = Get-AutomationVariable -Name 'VMName'
# Login to azure
Add-AzureRMAccount -Credential $Credential  -SubscriptionId $SubscriptionId
# Get the VM
$VM = Get-AzureRmVM -Name $VMName -ResourceGroupName $ResourceGroup

If ($VM.HardwareProfile.VmSize -eq $HWProfile)
 {
     Write-Output "HW size already set to $($VM.HardwareProfile.VmSize)"
 }
 Else
 {
     Write-Warning "Scaling up to $HWProfile"
	 # Set new VM size
     $VM.HardwareProfile.VmSize = $HWProfile
	 # Update VM
     Update-AzureRmVM -VM $VM -ResourceGroupName $ResourceGroup
     Write-Output "HW scaling up to $($VM.HardwareProfile.VmSize)"
 }

My Azure Automation variables look like this…

image

I hope this helps you for your next demo / PoC or what ever project you need it for.

ARM Template – Deployment Error “[Subscription().SubscriptionId]” The request is invalid…

Error

Recently I authored some ARM stuff in Visual Studio and I had a need for getting the current subscription ID the template is being deployed to.  So there is a helper function you can call like this…

“subscriptionId”: “[subscription().subscriptionId]”

I used it in the template like this…

2

…but as soon I tried to deploy the template, I hit this error all the time…

“code”: “BadRequest”, “message”: “{\”Message\”:\”The request is invalid.\”,\”ModelState\”:{\”variable.properties.value\”:[\”Invalid JSON primitive: 328de222-1a51-458a-96be-6770259e84c0.\”]}}”

3

I am not sure why this happen, but I figured out a workaround. If I concatenate the entire subscription id, it would work like this…

1

I hope this saves you some headache!

SCOM – Certificate Missing Enhanced Key Usage EventID 20050

missing

If you want to monitor a server which does not belong to a domain you need to use a certificate, which has special requirements. You will find many posts how to handle SCOM certificates using a Microsoft PKI on the internet. An example is the detailed post from Tyson Paul. One of the essential requirements for the certificates is to provide the Enhanced Key Usage properties for Client Authentication (OID 1.3.6.1.5.5.7.3.2) and Server Authentication (OID 1.3.6.1.5.5.7.3.1). If you do not provide these properties you will receive an error in the Operations Manager event log…

image

A problem you could face in the real world is, that some customers won’t allow you to create the certificates for SCOM and they might have “generic” certificates for other use cases. Usually YOU provide the request file and provide the configuration for the certificates. Under certain circumstances this might not be the case. This means, that you might certain properties will be missing on the certificate itself. In case of SCOM, you can add the missing properties on the certificate. Just go to the Details of the certificate after you imported it into your computer. Click Edit Properties and select the purpose in the dialog, like this…

image

Having this option in place, let’s you successfully monitor the workgroup servers.

This will probably save you some headache 🙂 .

OMS – Disconnect Azure Storage Account from Workspace

Whereisit

In OMS you are able to collect data from storage account? Why is this useful? Well, there times where you want to store data from different Azure sources for a longer time than provided by Azure itself and then dig into the data using OMS. For example you are able to store IIS Logs, Windows Events, Syslog (Linux), Windows Tracing Logs (ETW Logs) or Service Fabric Events. In the past days you could just configure the settings within the OMS portal itself.

StorageAccount

In the current OMS portal you simply see something like this…

image

…the documentation link does not provide much help in terms of connecting or removing these accounts. Therefore go to the new Azure portal, select your workspace and select “Storage account logs” and click Add

Continue reading

SCOM 2016 – Upgrade Notes from the Field

upgrade

Upgrading from SCOM 2012 R2 to SCOM 2016 is theoretically no such big deal. BUT sometimes you could face issues at the customer’s infrastructure, which force you to take some extra hurdle. This post should give you a high level overview of different migration scenarios and additionally some pitfalls you could meet upgrading to SCOM 2016.

High level upgrade path

There are 3 ways to upgrade a SCOM 2012 R2 environment.

1. Side-by-side migration (“Slow Motion”)

image

  • This is probably the way which has almost no risks, but takes a long time to finish and has a consequence that you loose old data. Why is this? You install a brand new SCOM 2016 management group, having brand new databases (OperationsManager / OperationsManagerDW / OperationsManagerAC). If needed you also install separate Web Console, Reporting and if needed the ACS role also on a dedicated (management) server. I think the best option is to install all these SCOM 2016 roles on a brand new Windows Server 2016 server and the databases on SQL Server 2016. This way you have the latest and greatest technologies available and you are armed for the next couple of years. Having this in place you are able to dual-home (multi-homing) the agent which is sending data to both management groups SCOM 2012 R2 and SCOM 2016. There is a good article on TechNet Wiki how to configure multi-homing if you have multiple AD forests or here if you have agents deployed in the same AD forest. As soon you have the new management group up and running you need to migrate all management packs, channels, subscriptions, overrides, roles etc. There are ways to export and import this stuff, but I recommend if you are choosing this upgrade path, then I would start configuring SCOM from scratch. Especially creating new overrides and documenting them will give you a chance to have a well configured and documented SCOM environment. One huge advantage of this upgrade path is, that you are able to upgrade to new versions of existing management packs, implementing new management packs and testing them thoroughly with no impact on your production SCOM environment until you switch management group and turn on notifications. This approach has also few disadvantages:
  • It takes usually a long time to finish this migration.
  • There are 2 management groups to maintain.
  • The amount of work to tune the management packs should not be underestimated.
  • Dual-homing an agent could lead to some more stress on the agent server.

2. SCOM In-place only upgrade (“Big Bang”)

image

  • If you decide to go for an in-place upgrade you are taking a much faster but also “risky” path, which needs more pre-work, testing and in case of failures also some plans to revert the changes using backups and/or VM snapshots. An in-place upgrade is in theory not that much of a problem and also fully supported by Microsoft. The first step is to run the SCOM 2016 setup on a management server which will discover the roles on the management server and upgrade the server itself and also the SCOM databases to SCOM 2016. If you managed to successfully upgrade the first management server / management group, then you go for the next management server, ACS Collector, Gateways, Console, Web Console and Reporting Server. As soon you have upgraded all components you are all done. Sounds easy, but believe me, there are plenty of things that could fail. This approach has also few disadvantages:
  • Because you upgrade SCOM only, the operating system stays the same. Of course you could theoretically in-place upgrade the operating system as well, but I really don’t encourage you to do so. If you need to upgrade SCOM and the operating system as well, please check the next upgrade option.
  • All your SCOM configurations bad or good will stay. If your management group is badly configured it will stay badly configured – an upgrade won’t change anything.
  • You need to check if the management packs work with SCOM 2016, especially third party or community MP’s. Please ask the vendor BEFORE you start the upgrade.
  • Make sure you meet the system requirements for SCOM 2016 .
  • Remember there are also 3rd party connectors in SCOM which might are not supported by SCOM 2016.

3. SCOM In-place upgrade and OS upgrade (“Big Bang++”)

image

  • If you decide to go for an in-place upgrade and you also want to upgrade the operating system to Windows Server 2016 in your environment, then this is an elegant way to achieve this goal. The risks are the same as “in-place only” upgrade but in addition you need to have a good plan how to switch the SCOM agents and ACS Forwarders to the new management servers. Before you start upgrading, make sure you have new Windows Server 2016 servers installed, which will become the new management servers. Step 1 is to run in-place upgrade on an “old” SCOM 2012 R2 management server (make sure it meets SCOM 2016 system requirements). If this is finished upgrade the other SCOM 2012 R2 management servers to SCOM 2016 and also ACS Collector, Gateways, Console, Web Console and Reporting Server. Step 2 if your management group is upgraded successfully install SCOM 2016 management servers on the fresh installed Windows Server 2016 servers. Depending on your SCOM environment, but if you have ACS installed, you could also install ACS Collector on a additional dedicated SCOM 2016 management server running on Windows Server 2016. Step 3 move the Windows / Linux agents, ACS Forwarders to the new management servers / ACS Collector. Step 4 uninstall the “old” management servers from the management group. If you have Web Console and/or Reporting installed you could simply uninstall the features from the “old” SCOM servers and reinstall it on new Windows Server 2016 server pointing to the SCOM 2016 deployment. I recommend uninstalling Reporting and Web Console BEFORE you upgrade the management group. This scenario has the same problems as an “in-place only” scenario but additionally you have to be aware of few more things:
  • Switching the Windows / Linux agents or ACS Forwarders to the new management servers could take some time and depending on the amount of “clients” Step 3 needs to be planned carefully.
  • If you don’t have the agents controlled by the SCOM Console you need to prepare some PowerShell scripts for moving the agents / ACS Forwarders to the new management servers.
  • Remember to install certificates for Linux  or Windows agents monitoring on the new management servers.
  • Remember to set the SPNs for the new management servers.
  • If you changed settings (Registry)  on your old management servers, check if you need to make these settings on your new management servers as well.

Continue reading

SCOM 2016 – System.Data.SqlClient. SqlException, Exception Error Code: 0x80131904 Login failed for user

NoClue

Sometimes there are things you won’t understand. I tried to install a new SCOM 2016 management server in a virgin Azure VM (Windows Server 2016). The necessary SQL Server 2016  was placed on another Azure VM (Windows Server 2016). Well this is nothing special and I have installed SCOM a gazillion times, BUT now I faced this error…1

…of course I checked the setup log as suggested and I found this error…

[21:17:34]:    Error:    :Exception running sql string [NOT DEFINED]: Threw Exception.Type: System.Data.SqlClient.SqlException, Exception Error Code: 0x80131904, Exception.Message: Cannot open database “OperationsManager” requested by the login. The login failed.
Login failed for user ‘RETURNONE\Stefan’.
[21:17:34]:    Error:    :StackTrace:   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
    at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions

I double checked the permissions for my account, so it has sysadmin permission on the SQL Server but repeating the installation ended up in the same error. After a while I found the solution, I unregistered and registered msiexec program on the SCOM server like this…

image

…the I executed the setup again (make sure you always use elevation ;))and the installation succeeded. Thanks to Abu-Obaid who delivered the solution, all credits to him. I hope this will save you some time!

SCOM – Editing SPN Quickly

Edit

If you are dealing with SCOM, you know that there is a lot to install and configure before it runs smoothly. One step during the installation process, is to configure the SPN (Service Principal Names) in Active Directory. In fact you need to set SPNs per SCOM management server and if you are hosting the web console on a dedicated server you also need to set an SPN (and Kerberos constraint delegation) correctly, so authentication will work properly. But how should the SPNs look like? Well, Kevin Holman (Microsoft) published  while ago an awesome post how they should look like. It will answer probably most of your questions. If not, just drop me a comment 🙂 .

There are different ways to mess with SPNs settings. First born tool is SetSPN.exe, which has been around for a while and can be considered “classic”. A more modern way of doing SPN registration is to use PowerShell of course. In terms of SCOM, if you are using a domain account for System Center Data Access Service then you could use PowerShell cmdlet Set-ADUser to  to register SPNs. It would look like this…

Get-ADUser -Filter 'Name –eq "[DAS Account]"' | Set-ADUser -ServicePrincipalNames  @{Add="MSOMSdkSvc/[MgmtServerFQDN]"}
Get-ADUser -Filter 'Name –eq "[DAS Account]"' | Set-ADUser -ServicePrincipalNames  @{Add="MSOMSdkSvc/[MgmtServerNetBIOS]"}

But if you more into GUI’s or you need to troubleshoot quickly, it might be faster to use the Active Directory Users and Computers console. You need to turn on Advanced Features

image

…check the Attribute Editor on your System Center Data Access account and select the servicePrincipalName property…

image

There you have a quick and nice overview, what has been configured on you service account. In addition you are able to add and remove obsolete SPN’s. Hope this helps :).