Custom AlertManager in ROSA 4.9.x

This page is deprecated. In order to get the best experience for custom alerting in ROSA, please upgrade your cluster to to 4.12 and follow the newer documentation.

ROSA 4.9.x introduces a new way to provide custom AlertManager configuration to receive alerts from User Workload Management.

The OpenShift Administrator can use the Prometheus Operator to create a custom AlertManager resource and then use the AlertManagerConfig resource to configure User Workload Monitoring to use the custom AlertManager.

Prerequisites

• AWS CLI
• A Red Hat OpenShift for AWS (ROSA) cluster 4.9.0 or higher

Create Environment Variables

1. Before we get started we need to set some environment variables to be used throughout the guide.

Install Prometheus Operator

If you prefer you can do this from the Operator Hub in the cluster console itself.

1. Create a OperatorGroup and Subscription for the Prometheus Operator

Deploy AlertManager

1. Create an Alert Manager Configuration file

   This will create a basic AlertManager configuration to send alerts to a slack channel. Configuring slack is outside the scope of this document. Update the variables to suit your slack integration.

Configure User Workload Monitoring to use the custom AlertManager

1. Create an AlertManagerConfig for User Workload Monitoring

   Note: This next command assumes the existing config.yaml in the user-workload-monitoring-config config map is empty. You should verify it with kubectl get -n openshift-user-workload-monitoring cm user-workload-monitoring-config -o yaml and simply edit in the differences if its not.

Create an Example Alert

1. Verify it works by creating a Prometheus Rule that will fire off an alert

2. Forward a port to the alert manager service

3. Browse to http://localhost:9093/#/alerts to see the alert “ExampleAlert”

   

4. Check the Alert was sent to Slack