Actions

The alerting and response system in SentryOne uses the concept of Conditions and Actions. Actions can be defined in response to certain Conditions being met within your environment. You can choose from a variety of Actions, depending on which Condition is being responded to.

All Actions work on the principle of inheritance, meaning that if you configure an Action in response to a Condition being met at the Global level (All Targets), it's automatically passed down to all applicable objects below it. This allows you to define global Actions for the most common issues across your environment once, and have those passed down to every monitored server automatically. You can further refine Actions at each level as needed. For a visual representation of how inheritance works within SentryOne, see the Alerting and Response System Hierarchy diagram.

Each Action that's configured in your environment has an associated behavior. The behavior controls how the Action is carried out relative to any inherited Actions. There are three action behaviors available: 

  • Override Inherited Actions 
  • Combine with Inherited Actions  
  • Disabled 

For a complete explanation and example usage scenario for each behavior, see Action Behaviors below.

Configure Actions for a specific group of objects or an individual object to give you control over how SentryOne works throughout your environment. SentryOne comes with a number of global Actions predefined to get you up and running quickly. These Actions can be changed, as needed to fit the specific needs of your environment.

Response Rulesets control how often Actions are taken in response to Conditions being met. They do this by assigning additional criteria on top of the Condition itself, that must be met before an Action takes place. To control the time frame of when an Action is processed apply a Window directly to any configured Action.

Configuring Actions

If you're upgrading from a previous version of SentryOne, you'll notice that the Conditions and Settings pane has been updated.

The Conditions displayed in the Conditions pane change depending on which node or object you've selected in the Navigator pane. If you don't see the Conditions pane once you've selected a node in the Navigator pane, select View > Conditions.

SentryOne Conditions pane diagram

Select the All Targets node to see globally applied Conditions in the Conditions pane. 

SentryOne Conditions pane Global Conditions

When you select any applicable object level below the All Targets node, you'll see two specific sets of applied Conditions in the Conditions pane.

SentryOne Conditions pane Inherited conditions and Explicit conditions

The top section of the Conditions pane contains the Inherited section, which shows any applied Conditions that are being passed down to the current object level. The Inherited section also contains an Object column, which shows the object level the Condition is being inherited from. When an Inherited condition is overridden or disabled, it still shows up in the Inherited section, but its text is grayed-out, and its status says Overridden.  

SentryOne Conditions pane Overridden conditions

Directly beneath the Inherited section, is the Explicit section. The Explicit section contains applied Conditions that have been set at the current object level. The Explicit section also displays a Behavior column. Each Action that's set up in your environment has an associated behavior. This behavior controls how the Action is carried out relative to any inherited actions.

SentryOne Conditions pane Explicit conditions

Adding a New Action

  1. To add a new Action, select the desired node in the Navigator pane.
    SentryOne select the desired node in the Navigator
  2. Select Add in the Conditions pane (View > Conditions) . This opens the Action Selector window.
    SentryOne Conditons pane Add Explicit conditions
  3. Expand the applicable object and Condition. Use the check box(s) to select which Actions should be taken in response to this Condition being met. Select OK to save your changes.
    SentryOne Actions Selector

Note:  You may also choose to quickly Disable, Override, or Combine any inherited actions. To do so, select a Condition in the Inherited section of the Conditions pane and choose the desired command (Disable , Override, or Combine).

SentryOne Inherited conditions toolbar

Action Behaviors

Each action that's setup in your environment has an associated behavior. This behavior controls how the action is carried out relative to any inherited actions.    

The user can change the Behavior of any Action to one of the following:

BehaviorDescription
Override Inherited Actions

Actions defined with this behavior override all actions that are being inherited from a higher level. This Behavior acts as a special set of instructions that are followed instead of the passed down(inherited) instructions.

When a Condition occurs that triggers an action defined with the Override Inherited Actions behavior, inherited actions aren't executed, but the action that's defined at that specific level is executed.

The Override Inherited Actions behavior is the default behavior assigned for any newly defined action.

For an example scenario, see the Override Inherited Actions Example.

Combine with Inherited Actions

Actions defined with this behavior combine with all inherited actions. This behavior acts as a set of instructions that are followed in addition to the passed down(inherited) instructions.

When a Condition occurs that triggers an Action defined with the Combine with Inherited Actions behavior, all inherited actions execute, and all actions that are defined at that specific level also execute.

For an example scenario, see the Combine with Inherited Actions Example.

Disabled

When an action is defined with the Disabled behavior it disables any action of the same action type that's being inherited for a certain Condition. This behavior acts as a special set of instructions that disallow the passed down (inherited) set of instructions.

When a Condition occurs that triggers an Action defined with the Disabled behavior, inherited actions aren't executed.

For an example scenario, see the Disabled Example.

Multiple Actions per Condition

One of the new improvements made to the alerting and response system is the ability to add multiple versions of the same action type per condition. Use this ability, in combination with Response Rulesets to implement escalation procedures when certain Conditions take place.

Note:  Configured actions of the same action type, defined at the same object level, have their behavior types bound together because inherited actions can't simultaneously be in more than one state (Disabled, Combined, and Overridden).

Note:  For any condition you can choose multiple actions. There are 11 actions available for every condition.

Examples

Override Inherited Actions Example

Actions defined with the Override Inherited Actions behavior override all actions that are being inherited from a higher level. This behavior acts as a special set of instructions that are followed instead of the passed down(inherited) instructions.

For this example assume the following:

There's a Send Email action configured at the Global level that has a selected target of DBA1 for the SQL Server Agent Job: Failure condition.

With this Action configured as above, anytime a SQL Sever Agent Job fails across the entire monitored enterprise the following will occur:

  1. The SQL Server Agent Job: Failure condition is met.
  2. In response to the Condition being met, the Send Email action executes, sending a notification email to DBA1.

Now suppose there's a development server (DEVServer) in the environment, that's not managed by DBA1. For this server, when an Agent Job fails, you'd like notification emails sent to the developer or user DEV1 instead of the user DBA1 being alerted.

To accomplish this, add a Send Email action with a selected target of DEV1, for the SQL Server Agent Job: Failure condition at the instance level with a behavior of Override Inherited Actions.

With this Action configured as above, anytime a SQL Sever Agent Job fails on the development server the following occurs:

  1. The SQL Server Agent Job: Failure condition is met.
  2. In response to the condition being met, the Send Email action executes, sending a notification email to DEV1.

Only the development server is impacted by this change. The rest of the monitored enterprise is still subject to the globally configured Send Email action that has a selected target of DBA1.

Combine with Inherited Actions Example

Actions defined with the Combine with Inherited Actions behavior are combined with all inherited actions. This behavior acts as a set of instructions that are followed in addition to the passed down(inherited) instructions.

For this example assume the following:

There's a Send Email action configured at the global level that has a selected target of DBA1 for the SQL Server: Offline condition.

With this Action configured as above, anytime a SQL Sever is detected to be offline across the entire monitored enterprise, the following occurs:

  1. The SQL Server: Offline condition is met.
  2. In response to the Condition being met, the Send Email action executes, sending a notification email to DBA1.

Now suppose there's a SQL Server (ASPServer) in the environment that has an internet facing workload. When the SQL Server: Offline condition is met you'd still like to notify DBA1, but you'd also like to notify the web developer, WEBDEV1.

To accomplish this, add a Send Email action, with a selected target of WEBDEV1, for the SQL Server: Offline condition at the instance level with a behavior of Combine with Inherited Actions.

With this Action configured as above, anytime that the ASPServer is detected to be offline, the following occurs:

  1. The SQL Server: Offline condition is met.
  2. In response to the condition being met, the Send Email action executes, sending a notification email to DBA1 and WEBDEV1.

Only the ASPServer is impacted by this change. The rest of the monitored enterprise is still be subject to the globally configured Send Email action for the SQL Server: Offline condition, that has a selected target of DBA1.

Disabled Behavior Example

When an Action is configured for a Condition with the Disabled action behavior, it disables any action of the same action type that's being inherited. This behavior acts as a special set of instructions that simply disallow the passed down(inherited) set of instructions.

For this example assume the following:

On TestServer1 there's a Send Email action configured that has a selected target of DBA1 for the SQL Server Agent Job: Runtime Threshold Max condition. This action is configured at the instance level.

With this Action configured as above, anytime any SQL Agent Job meets the Runtime Threshold Max condition on TestServer1 the following occurs:

  1. The SQL Server Agent Job: Runtime Threshold Max condition is met.
  2. In response to the Condition being met, the Send Email action executes, sending a notification email to DBA1.

Now suppose there's a job, LongRunningJob, that's known to be long running on TestServer1. DBA1 doesn't need to be alerted about it and would like to disable runtime notifications about this job.

To accomplish this, add a Send Email action for the SQL Server Agent Job: Runtime Threshold Max condition at the object level(the LongRunningJob) with a behavior of Disabled.

With this Action configured as above when the LongRunningJob runs long, the following occurs:

  1. The SQL Server Agent Job: Runtime Threshold Max condition is met.
  2. In response to the Condition being met, the inherited Send Email action isn't executed.

Only the individual SQL Agent Job LongRunningJob is impacted by this change. The rest of the SQL Agent Jobs on TestServer1 are still subject to the instance level Send Email action configured for the SQL Server Agent Job: Runtime Threshold Max condition.

Available Actions

Kill Task

The Kill Task action kills, or cancels the current job. This action is typically associated with Runtime Maximum, Performance Threshold Maximum, Block, or Conflict conditions, to prevent resource contention situations. 

SentryOne Kill Task action

Note:  It's not applicable to the Started condition, or any conditions implying event instance completion.

Log to Windows Event Log

The log to Event Log action sends data relevant to the condition to the application Event Log on the SentryOne monitoring service computer.

SentryOne Log to Windows Event Log action

Settings

Within the Actions Settings tab, select the Log type ( Information, Warning, or Error) and then assign a Log ID.
SentryOne Log to Windows Event Log Actions Settings

Log to Disk

The Log to Disk action sends data relevant to the condition to the specified text file on the SentryOne monitoring service computer's file system.

SentryOne Log To Disk action

Settings

Within the Actions Settings tab, enter the Log file where you want to store the condition data.
SentryOne Log to Disk Action Settings

Send Email

The Send Email action sends an email alert to the Users or Groups specified in the Action Settings. For more information about Users and Groups, see the Contact Management topic. Certain alert emails in version 7.0 now include a View in SentryOne Client section. You'll notice that the Event View section contains a hyperlink that begins with the SentryOne URL protocol. Selecting this link opens the Event View in the SentryOne client for the specific condition that triggered the alert.

SentryOne Send Email action

Note:  Stop receiving the Outlook Security Notice when you select an alert email link by enabling trust for the SentryOne protocol. Enable the SentryOne protocol by entering the appropriate key in the Windows Registry Editor: 

For Outlook 2010

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\14.0\Common\Security\Trusted Protocols\All Applications\sqlsentry:]

For Outlook 2013

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Common\Security\Trusted Protocols\All Applications\sqlsentry:]

For Outlook 2015 or higher

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Common\Security\Trusted Protocols\All Applications\sqlsentry:]

For more information about enabling trust to a specified protocol, see this MSDN article.

Settings

The Actions Settings tab allows the following configurations:

OptionDescription
Select TargetsSpecifies which Users or Groups receive the email alert.
ImportanceSpecifies the importance level of the email alert.
From AddressSpecifies the from email address of the alert. To use the default email address leave this blank. For more information, see the SentryOne Monitoring Service Settings topic.

SentryOne Conditions pane Action Settings Select Targets

Execute SQL

The Execute SQL action executes the T-SQL statement(s) on the specified SQL Server.

SentryOne Execute SQL action

Settings

Use the server list to select a server, and enter the desired T-SQL in the command text box. Selecting the Target option executes the T-SQL statement against the server that triggered the condition.

SentryOne Conditions pane Action Settings Execute SQL

Note:  In an environment with multiple monitoring services, it's important to note that the service that detects the condition is the service that fires the action. If that service doesn't have connectivity to the targeted server, the action fails.

Send SNMP Trap

The Send SNMP Trap action sends a Simple Network Management Protocol (SNMP) trap notification. For more information about configuring SNMP, see the SentryOne Monitoring Service Settings topic.

SentryOne Send SNMP Trap action

Note:  Locate the files for configuring SNMP MIBs in the SentryOne application folder C:\Program Files\SentryOne\18.0\SNMP MIBs.

SentryOne SNMP MIBs

Log To Database

The Log To Database action logs data relevant to the condition to the Actions log in the SentryOne database.

SentryOne Log To Database action

Note:  Any of the other actions also write to the database, so if you’ve chosen another action this may be redundant.

Execute Process

The Execute Process action executes the defined command text on the specified SQL Server.

SentryOne Execute Process action

Settings

Within the Execute Process Action Settings tab you'll see the Execute Process section. Use the server list to select a server. Selecting the Target option from the server list executes the process against the server that triggered the condition. Use the Command Text box to enter the desired literal command.

SentryOne Conditions pane Action Settings Execute Process

Note:  Pass certain  system parameters, as well as user defined parameters through output text.

Send Page

The Send Page action sends an alert to the pager address for the specified Users or Groups. For more information about Users and Groups, see the Contacts Management topic.

SentryOne Send Page action

Settings

The Actions Settings tab for the Send Page action allows the following configurations:

OptionDescription
Select TargetsSpecifies which Users or Groups receive the pager alert.
ImportanceSpecifies the importance level of the pager alert.
From AddressSpecifies the From address of the alert. To use the default email address leave this blank. For more information see the SMTP configuration section of the SentryOne Monitoring Service 
Settings topic.

SentryOne Conditions pane Actions Settings Select Targets

Run QuickTrace™

The Run QuickTrace action executes a Quick Trace™ against a specified target server. A Quick Trace is a comprehensive snapshot of activity created by combining process-level data and trace events collected during a brief sample period. A Quick Trace isn't filtered, so it collects all events on the specified SQL Server.
SentryOne Run QuickTrace Action

Note:  The QuickTrace action is now available for a wide range of conditions.

To avoid impacting the performance of the target SQL Server, Quick Trace is time and size limited. Only one simultaneous Quick Trace can be run against any target server. Quick Traces are throttled to control the time between successive runs. The current allowed frequency is one minute, meaning that once a Quick Trace is completed, another Quick Trace can't begin on the same specified server until one minute has elapsed.

Note:  To avoid impacting server performance on very busy systems, there are certain cases where SentryOne further restricts Quick Trace functionality. If one of these cases is detected the Quick Trace action doesn't fire. For more information, see Quick Trace Restrictions.

Settings

The Actions Settings tab for the Quick Trace action allows the following configurations:

OptionDescription
Run QuickTrace AgainstSpecifies the Target server where the Quick Trace runs.
Run QuickTrace forSpecifies the length of time that the Quick Trace runs against the target server.
Collect Statement EventsSpecifies whether to collect Statement Level Events. Enabling the Collect Statement Events option dramatically increases the amount trace data collected.  
Limit Trace Data toSpecifies the maximum number of rows that the Quick Trace collects.

SentryOne Conditions pane Actions Settings Run QuickTrace Against

Execute Job

The Execute Job action executes a SQL agent job on the current server, or any other watched server in your enterprise. This action enables you to create one-to-one job chains by associating it with Completed, Success or Failure conditions. For more advanced chaining features, see Event Chains.

SentryOne Execute Job action

Settings

Within the Action Settings tab you'll see the Jobs To Execute section. Select a server from the server list. Specify the SQL agent job to run by using the job list. Once a target job has been specified for this action, it shows up in the Event Calendar view with a grid pattern background as an indication of its conditional execution.

SentryOne Conditions pane Actions Settings Job to Execute

Execute PowerShell

The Execute PowerShell Action executes the entered PowerShell script on the monitoring computer, the target computer, or any valid Windows target.

SentryOne Execute PowerShell action

Note:  You may also use system parameters in the PowerShell script, with a format of <%ParameterName%> (e.g, $ServerName = '<%ServerName%>').

Settings

Within the Action Settings tab you'll see the Server section. Select the server where the script will be executed from the server list (on the monitoring server, on the target server, or on a specific watched server).Select whether the defined PowerShell Execution Account, a domain account, or a local account runs the script from the account list. 

Note:  A valid user name and password must be provided in the Action Settings if the domain account or local account options are selected.

Enter or load (selecting  Load from File beneath the text area) PowerShell script text into the PowerShell Script Text section. Select Test PoSh Script to test the entered script on a specified target.
SentryOne Conditions pane Action Settings Execute PowerShell

Send To Alerting Channels

The Send to Alerting Channels action allows open Advisory Condition events to display in various places throughout the client.

Selecting the Performance Analysis Dashboard option shows the open Advisory Condition events on the Performance Analysis Dashboard provided that the Advisory Condition involves a performance counter that's associated with one of the charts on the Dashboard. A glyph also appears in the top left corner of the chart and provides additional information pertaining to the Advisory Condition event.

Select a user to be automatically assigned to an Advisory Condition event by selecting a user from the dropdown list that appears in the Action Settings tab of the Conditions pane.