Reset Monitor using SCOM 2012 and Orchestrator – A must have Runbook

There is at least one Runbook each SCOM responsible must have. Are you wondering which? Well, I am sure you are always aware of what kind of alert you are closing. Is this alert beeing generated by a rule or by a monitor. If it is generated by a rule you just can close the alert and you don’t have to care about anything. But if the alert is being generated by a monitor you always facing the problem that if you just close the alert itself your monitor will often stay in a critical or warning state. A good practice to close alerts from monitors is to fix the problem itself in your IT infrastructure and then go to the health explorer and reset the monitor. After that the alert will also close.

A similar problem occurs in a more advanced situation. Let’s assume System Center Operations Manager is generating incident tickets in System Center Service Manager through the connector. After the ticket has been closed in Service Manager the alert in Operations Manager will also close. But what if the alert has been generated by a monitor? Well, Service Manager does not reset the monitors in Operations Manager.

For both situation you could build a Runbook in System Center Orchestrator Smiley.

It looks like that….simple huh?

image

O.k. let me explain. The first activity “Monitor Alert” is constantly monitoring the alerts for updates. If an alert is updating its status from “New” to “Closed” e.g. you close an alert using the SCOM console or Service Manager closes the alert. Each time the Runbook will start. The “Monitor Alert” activity will pass the ID from the alert to the“Reset Monitor” activity and then run/trigger the script activity. This script will reset the monitor which has generated the alert. Easy? Yes.

Imagine it does not matter if you are closing the alert or Service Manager will close the alert in both situations if the alert was generated by a monitor the Runbook will do its work.

The “Monitor Alert” activity looks like this…

image

…and the “Reset Monitor” activity and script which I slightly modified from here looks like this…

image

Script Code:

# Import Operations Manager Module and create Connection
Import-Module OperationsManager;
New-SCOMManagementGroupConnection scom01;
# Get alert
$alertid=”Put here published data ID from “Monitor Alert” acitivity]“;
$alerts = Get-SCOMAlert | where { $_.id -eq $alertid};
foreach ($alert in $alerts)
{
# Get IDs
$mrid = $alert.monitoringruleid;
$mcid = $alert.monitoringclassid;
$moid = $alert.monitoringobjectid;
# Get Objects
$monitor = Get-SCOMMonitor | where {$_.id -eq $mrid};
$monitoringclass = Get-SCOMClass | where {$_.id -eq $mcid};
$monitoringobject = Get-SCOMMonitoringobject -class $monitoringclass | where {$_.id -eq $moid};
# Reset Monitor
$monitoringobject | foreach{$_.ResetMonitoringState($monitor)};
};

Make sure you modify line 5 with the published data!

Now every time the monitor gets reset properly and you don’t have to care about it anymore.

Cool, SCOM and SCORCH better together!

50 Comments

      • Stefan,

        I’ve been playing around with Orchestrator over the last few days and went to implement your runbook above.

        When I either test the runbook or simply just run it, I get a failure on the ‘Run .Net Script’ action with an error alert like this:

        File C:\Program Files\System Center Operations Manager 2012\Powershell\OperationsManager\OperationsManager.psm1 cannot be loaded because the execution of scripts is disabled on this system. Please see “get-help about_signing” for more details.

        Now, I know you might be thinking that I haven’t set my execution policy to either ‘RemoteSigned’ or ‘Unrestricted’, but this was the first thing that I did on both the SCOM and SCOrch servers and when I run ‘get-executionpolicy’ on both servers, it comes back with ‘Unrestricted’. I’ve tried rebooting both servers and also recreating the runbook again.

        Any ideas for someone whose PoSh Kung Fu is pretty weak?

        Thanks!

      • Hi Kevin,

        Cool, great hearing from you.

        Try to start the Powershell in “C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe” on your Orchestrator server and set there also Set-Executionpolicy unrestricted ;).

        Let me know if it worked.

        Regards,

        Stefan

  1. Stefan, that sorted the problem out straight away – I should’ve known it was something as simple as that – was trying to debug the issue a little too deep when I hadn’t thought about the WOW64 PowerShell.exe instance :)

    I’ll know the next time I have this issue for sure now!

    Thank you so much for the rapid response and excellent Orchestrator runbooks – it’s making my ramp-up on it much easier!

    Kevin.

  2. Stefan,
    Thanks for the great post as this will truly be helpful. HOwever, I’m having the following issue using SCOM 2012 and Orchestrator 2012. Using the following script;
    # Import Operations Manager Module and create Connection
    Import-Module OperationsManager;
    New-SCOMManagementGroupConnection ;
    # Get alert
    $alertid=”{Id from “Monitor Alert”}”;
    $alerts = Get-SCOMAlert | where { $_.id -eq $alertid};
    foreach ($alert in $alerts)
    {
    # Get IDs
    $mrid = $alert.monitoringruleid;
    $mcid = $alert.monitoringclassid;
    $moid = $alert.monitoringobjectid;
    # Get Objects
    $monitor = Get-SCOMMonitor | where {$_.id -eq $mrid};
    $monitoringclass = Get-SCOMClass | where {$_.id -eq $mcid};
    $monitoringobject = Get-SCOMMonitoringobject -class $monitoringclass | where {$_.id -eq $moid};
    # Reset Monitor
    $monitoringobject | foreach{$_.ResetMonitoringState($monitor)};
    };

    I get a SUCCESSFUL run when using the RUNBOOK tester, but when actually running the script I get the following error in my Orchestrator Log History;

    Run.NET script – FAILED
    Error Summary Text
    “The term ‘New-SCOMManagementGroupConnection’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.”

    Any idea why this is not working for me? Typing the same commands from my shell both x86 and x64 versions work fine???

    • Update: also see this in event log.

      C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe
      2544

      Microsoft.EnterpriseManagement.Core.Cmdlets, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

      System.DllNotFoundException: Unable to load DLL ‘MOMBIDldr.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
      at Bid.internalInitialize()

      • Hi,

        I have seen this error before in other situations.

        Sometimes it helps to copy the ‘MOMBIDldr.dll’ to copy into the mentioned directory here C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe. But please test it before going into production.

        Regards,

        Stefan
        .

      • Thanks Stefan, I had already done that. What I just found out is that Orchestrator ‘requires’ a reboot after installing the OpsMgr shell/console even though it does not ‘require’ a reboot :) Guess it can’t load the .dll without the reboot. Just rebooted and it worked like a champ. Thanks again for the great article, look forward to seeing more in the future. Hope this helps someone else!

  3. Hi Stefan,
    Thanks for the great post as this will truly be helpful.

    My question is:
    After the monitor alert capture an alert the Runbook is finish.
    I want to continue monitoring new alert.
    What would be the best way to handle it?
    What happen if we got two alert at the same time (we have a big environment – around 600 servers) ?

    Thanks a lot for your great job.
    Gilad

    • Hi Gilad

      Your welcome, glad I could help.

      The first activity of the runbook is already constantly runnig and “monitoring” for new alerts comming in. Unless you are stopping the runbook there is no need to do anything. The “Monitor Alert” activity is always “on”…

      Orchestrator will handle this for you, all new alerts will be processed separately you don’t have to be concernt about it. I am using this runbook also in a environment of ~600 servers.

      Conclusion: Just build it and test it, there is nothing that can go wrong….

      I hope this helps

      Stefan

  4. Hi Stefan,

    I’m using the Runbook and we see something strange.
    you may be able to help me with that.
    I see in the log that the trigger for the monitoring alert is activate every few second.
    i don’t now if there is a problem in SCOM or in ORCHESTRATOR but its cause us a very heavy traffic.
    i also see that we got the same alert every few second.
    i wrote the output to the log.

    This is example of the log:

    <1The monitor has been initialized for the first time or it has exited maintenance mode>
    <1The monitor has been initialized for the first time or it has exited maintenance mode>

    us you can see we got the same alert every ten second. (its only example of 2 alert but we got a lot of alerts)

    Do you know what is the problem?
    Thanks for your help,
    Gilad

  5. Pingback: Create a Orchestrator Runbook to reset SCOM objects to “healthy” « IT Hassle

  6. Hi,
    Thanks for this useful runbook example! I implemented it yesterday but encountered a problem:
    Each time an alert is processed by the runbook (Powershell) and is resetted, the “Monitor Alert” discovers it again and again (as for SCORCH, it seems to be an updated alert). I’ve tried to add the filter “MonitoringObjectHealthState” does not equal “Healthy”, but it did not work as expected. Do you have any recommendations?
    By the way, I haven’t used the “Run .NET Activity” but the “Execute PS Script” from Codeplex IP, as I wanted to run it as a different user than the SCORCH service account.
    Thanks in advance!

      • Hi!

        I managed to fix Klaus’ problem. I added an extra activity to update a customfield in the alert. This results in the parameter LastModifiedBy of the alert beeing edited to the account that SCO uses to connect to SCOM.

        If you then add “LastModifiedBy does not equal … (the same account)” to the Monitor Alert filter, SCO will discover the alert only once.

        This offcourse only counts if the alert doesn’t get closed by SCO in the first place (with that account).

        Regards,
        Matthias

  7. Pingback: Orchestrator 2012: Reset SCOM 2012 monitor for closed alert | SystemCenterTipps

  8. Pingback: bad OM Performance because State Change Events are not groomed out - Mihai Sarbulescu's System Center Blog - Site Home - TechNet Blogs

  9. Hi Matthias, Stefan,

    I could configure the script like you said, i added to the alert a CustomField: Closed by Orchestrator
    The only thing I Need to figure out is that i have a lot of warnings with following value:

    Cannot validate argument on parameter ‘Class’. The argument is null or empty. Supply an argument that is not null or empty and then try the command again.

    $monitoringobject = Get-SCOMMonitoringobject -class $monitoringclass | where {$_.id -eq $moid};
    I guess the error is an empty value at $monitoringclass.

    Can you help me! Maybe we can Change the script to say if $Monitoringclass empty go to next….

    Regards,
    Martin

  10. Pingback: Excellent runbook for SCOM 2012 – In my head...

  11. Arne,
    I had the same issue. The problem in my case is/was that the executed powershell ran in an environment where the OperationsManager module was not registered and I was therefore not able to do import-module OperationsManager. However, I created a powershell script which I put on all Management Servers (c:\users\public\documents\scripts\scomps.ps1) and which is “included” in all powershell command channels
    scomps.ps1:
    Import-Module “C:\Program Files\System Center 2012\Operations Manager\Powershell\OperationsManager\OperationsManager.psm1″
    . “C:\Program Files\System Center 2012\Operations Manager\Powershell\OperationsManager\Startup.ps1″
    . “C:\Program Files\System Center 2012\Operations Manager\Powershell\OperationsManager\Functions.ps1″

    and my command channel is as followed:

    Full path to cmd:
    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

    CMD parameters (example where I set a special resolution state to grab it via Orchestrator):
    . c:\users\public\documents\scripts\SCOMPS.ps1; Get-SCOMAlert -Id (‘$Data/Context/DataItem/AlertId$’) | Set-SCOMAlert -ResolutionState 13;

    Startup folder:
    C:\Windows\System32\WindowsPowerShell\v1.0\

  12. Has anyone seen the issue for the line
    $monitoringobject = Get-SCOMMonitoringobject -class $monitoringclass | where {$_.id -eq $moid}
    that results in high memory consumption and never completes when the object is an interface on a network device? I see the following event id 26373 on the SCOM management server:
    The query resulted in 230773 rows.
    This may cause high memory consumption, and affect server performance.
    Consider using buffered mode for this query.

  13. Stefan,

    This is an excellent runbook!
    I have added a little thing in our situation. This runbook reset also monitors for alerts that were closed by the system. We don’t need to reset this ones as well so I added to the scomalerts def:
    | where {$_.resolvedby -ne “System”}

  14. Pingback: SCOM 2012 – MAS Tool V 1.0 | stefanroth.net

  15. The Script can be faster and easier: The Runbook starts the Powershell-Script once for each Alert.
    So, you don’t need the for-each-loop. All used Commandlets accept Id as Parameter so you don’t need to querry all Objects and Pipe it to where {$_Id -eq …). With this enhancements no MonitoringClass-Instance is needed. And last but not least Get-SCOMMonitringobject returns only one Instance. Again there is no for-each needed.

    # Import Operations Manager Module and create Connection
    Import-Module OperationsManager;
    New-SCOMManagementGroupConnection hgf-int-vm-273
    # Get alert

    $alert = Get-SCOMAlert -Id “\`d.T.~Ed/{A28FF5FE-996F-4211-84A8-16836A554471}.Id\`d.T.~Ed/”

    # Get IDs
    $mrid = $alert.monitoringruleid
    $moid = $alert.monitoringobjectid

    # Get Objects
    $monitor = Get-SCOMMonitor -Id $mrid
    $monitoringobject = Get-SCOMMonitoringobject -Id $moid

    # Reset Monitor
    $monitoringobject.ResetMonitoringState($monitor)

  16. Hello Stefan,

    thanks for this helpful Runbook. I have one Question:
    I am getting the error that my user has not enough rights to do the resolve. It’s my SCO User. I don’t finde the possibility to change the user. So should I give the SCO user rights at SCOM?

    Regards, Doreen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s