Notifications
Customers can optionally configure the platform to automatically send "notifications" when Alerts are generated (i.e for both alerts being opened and closed).
Notifications can be configured to be sent to a list of "email addresses" that are specified and managed by the Org Admins. Alternately, customers can also customize the Alert Manager add-on which is part of the default cluster blueprint to send the notifications directly to 3rd party systems such as "Slack", "Pager Duty", "Ops Genie" etc.
Opt In Clusters¶
By default, email based notifications are disabled for all clusters under management. Administrators need to explicitly enable clusters for which they would like to receive email based notifications from the Controller.
- Click on Notifications on the cluster card
- Toggle notifications to enabled
Email Notifications¶
It is strongly recommended that customers configure a monitored email address for notifications. It may also be useful to implement customized email handling logic and workflows to process the incoming notifications.
Configuration¶
To manage the notifications at Organization level, perform the below steps:
- Navigate to System -> Alerts
- Click on Manage Notifications
- Enable/disable the required notification(s)
- Click Add Email to add one or more users to receive the selected alert notification(s)
- Click Save
Metadata¶
Email notifications are sent with meta-data in a specific format so that they can "programmatically parsed" by incident response systems as well as consumed by end users.
The meta-data in the email comprises "Key-Value" pairs that can be automatically parsed and used for downstream workflows. An illustrative example of a notification is shown below.
Customize Alert Manager¶
You can optionally customize and configure the Alert Manager "system addon" in the default cluster blueprint to send notifications to your existing incident response systems like Slack, Pager Duty, Ops Genie etc.
Step 1¶
- Go to Infrastructure > Addons
- Create a new addon and select the "Customize System Addon" option.
Step 2¶
- Create a new version for the custom system Alert Manager addon.
Step 3¶
- Download the default configuration manifests and customize them.
Step 4¶
- Upload the modified configuration to the addon
- Save the custom Alert Manager addon version
Step 5¶
- Go to Infrastructure > Blueprints
- Add this custom Alert Manager add-on to the custom blueprint for the cluster
Step 6¶
- Update the new blueprint version with the customized Alert Manager addon to the clusters
- The new system Alert Manager addon will be deployed to the clusters
After couple minutes, the blueprint will be published successfully and the customized Alert Manager addon will be deployed to the clusters.
Integration Recipes¶
Step-by-step recipes to customize the Alert Manager configuration for proactive notifications are documented below. Also, additional information on Alert Manager configuration is available.
| Integration Recipe |
|---|
| Slack |
| Pager Duty |
| Ops Genie |
| MSFT Teams |
| ServiceNow |