Prometheus creates and sends alerts to the Alertmanager which then sends notifications out to different receivers based on their labels. A receiver can be one of many integrations including: Slack, PagerDuty, email, or a custom integration via the generic webhook interface.
The notifications sent to receivers are constructed via templates. The Alertmanager comes with default templates but they can also be customized. To avoid confusion it's important to note that the Alertmanager templates differ from templating in Prometheus, however Prometheus templating also includes the templating in alert rule labels/annotations.
The Alertmanager's notification templates are based on the Go templating system. Note that some fields are evaluated as text, and others as HTML which will affect escaping.
Data is the structure passed to notification templates and webhook pushes.
|Defines the receiver's name that the notification will be sent to (slack, email etc.).
|Defined as firing if at least one alert is firing, otherwise resolved.
|List of all alert objects in this group (see below).
|The labels these alerts were grouped by.
|The labels common to all of the alerts.
|Set of common annotations to all of the alerts. Used for longer additional strings of information about the alert.
|Backlink to the Alertmanager that sent the notification.
Alerts type exposes functions for filtering alerts:
Alerts.Firing returns a list of currently firing alert objects in this group
Alerts.Resolved returns a list of resolved alert objects in this group
Alert holds one alert for notification templates.
|Defines whether or not the alert is resolved or currently firing.
|A set of labels to be attached to the alert.
|A set of annotations for the alert.
|The time the alert started firing. If omitted, the current time is assigned by the Alertmanager.
|Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.
|A backlink which identifies the causing entity of this alert.
|Fingerprint that can be used to identify the alert.
KV is a set of key/value string pairs used to represent labels and annotations.
type KV map[string]string
Annotation example containing two annotations:
summary: "alert summary",
description: "alert description",
In addition to direct access of data (labels and annotations) stored as KV, there are also methods for sorting, removing, and viewing the LabelSets:
|Pairs (list of key/value string pairs.)
|Returns a sorted list of key/value pairs.
|Returns a copy of the key/value map without the given keys.
|Returns the names of the label names in the LabelSet.
|Returns a list of the values in the LabelSet.
Note the default functions also provided by Go templating.
|strings.Title, capitalises first character of each word.
|strings.ToUpper, converts all characters to upper case.
|strings.ToLower, converts all characters to lower case.
|strings.TrimSpace, removes leading and trailing white spaces.
|Regexp.MatchString. Match a string using Regexp.
|pattern, replacement, text
|Regexp.ReplaceAllString Regexp substitution, unanchored.
|sep string, s string
|strings.Join, concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string. (note: argument order inverted for easier pipelining in templates.)
|html/template.HTML, Marks string as HTML not requiring auto-escaping.
|Returns the passed strings as a slice of strings.
This documentation is open-source. Please help improve it by filing issues or pull requests.