ServiceNow

Although a turnkey integration is available for email based notifications, organizations may wish to send notifications to other platforms in use internally. This recipe documents how organizations can customize the "Alert Manager" system add-on configuration and send the alerts to ServiceNow.

sequenceDiagram
    AlertManager->>+ServiceNow: Alert on cluster

Watch a video of this in action

---

Before you begin

• Ensure that the Event Management Connectors (sn_em_connector) plugin is installed on the Now Platform instance. They get installed as part of Event Management
• Verify Configuration Items for the hosts managed by Prometheus exist in ServiceNow. These CIs can be physical or virtual, and can be either manually created or discovered using IP discovery or Cloud discovery. For this exercise, Cloud discovery was used to populate the ServiceNow CMDB
• Role required: evt_mgmt_integration

Step 1: Configure ServiceNow

The first step in setting up the integration is to create a Push Connector Instance which uses the provided Prometheus Push Connector.

• Go to your ServiceNow instance and navigate to Event Management -> Push Connector Instances and select New to create a new record
• Enter a name, description, and select Prometheus Push Connector for the Push Connector Definition
• Click on Submit

• Copy the URL for the newly created connector, we will use this later as the receiver for Prometheus alerts

• Navigate to User Administration -> Users and select New to create a new record
• Enter a User ID, First name, Last name, Title, etc.
• Web service access only and Active should be selected
• Assign the evt_mgmt_integration role to the new User
• Click on Set Password to set the password and record for use later
• Click on Update to update the record

Step 2: Configure Alert Manager Add-on

Follow the below steps to create a customized "Alert Manager" System Add-on

• Navigate to Infrastructure -> Add-ons
• Create a new add-on using the Customize System Add-On option

• Create a new version for the custom system Alert Manager add-on

• Download the default configuration manifest (alertmanager-configuration.yaml)

• Add another webhook config under the receivers section of the downloaded configuration as shown below
• Add the username and password of the User account we created in ServiceNow
• Add the URL for the Push Connector Instance we created in ServiceNow

receivers:
- name: default-receiver
- name: rafay-webhook
  webhook_configs:
  - http_config:
      tls_config:
        ca_file: /etc/config/certs/ca.crt
        cert_file: /etc/config/certs/client.crt
        insecure_skip_verify: true
        key_file: /etc/config/certs/client.key
        server_name: '{{{ .V2EventFrameworkFQDN }}}'
    url: https://{{{ .V2EventFrameworkFQDN }}}/event/v1/alert/raise
- name: servicenow-webhook
  webhook_configs:
  - http_config:
      basic_auth:
        password: 0=2cC$+[3bi$_ITuUI
        username: rafayprometheus
    url: https://dev113070.service-now.com/api/sn_em_connector/em/inbound_event?source=prometheus&sys_id=99884718470c315088f5c561236d430a

• Next add the custom receiver for Prometheus alerts to the routes configuration in "route" section

route:
  group_interval: 5m
  group_wait: 10s
  receiver: rafay-webhook
  repeat_interval: 3h
  routes:
  - continue: true
    match_re:
      severity: .*
    receiver: rafay-webhook
  - continue: true
    match_re:
      severity: .*
    receiver: servicenow-webhook

• Upload the modified configuration to the add-on
• Save the custom Alert Manager add-on version

Step 3: Update Cluster Blueprint

Follow the below steps to deploy the customized Alert Manager configuration for ServiceNow notifications

• Go to Infrastructure > Blueprints and click on New Blueprint
• Give the Blueprint a name and click on Save

• Give the newly created Blueprint a Version Name
• Add the custom Alert Manager add-on to the custom blueprint and save changes

• Update the cluster with the new blueprint version that includes the alert-manager add-ons
• Click on Save and Publish

Step 4: Verify Configuration

Once the alerts are generated in the clusters, you will receive notifications in the Alerts section of your ServiceNow instance. An illustrative example is shown below.