<scrape_config>
<tls_config>
<azure_sd_config>
<consul_sd_config>
<dns_sd_config>
<ec2_sd_config>
<openstack_sd_config>
<file_sd_config>
<gce_sd_config>
<kubernetes_sd_config>
<marathon_sd_config>
<nerve_sd_config>
<serverset_sd_config>
<triton_sd_config>
<static_config>
<relabel_config>
<metric_relabel_configs>
<alert_relabel_configs>
<alertmanager_config>
<remote_write>
<remote_read>
Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc.), the configuration file defines everything related to scraping jobs and their instances, as well as which rule files to load.
To view all available command-line flags, run prometheus -h
.
Prometheus can reload its configuration at runtime. If the new configuration
is not well-formed, the changes will not be applied.
A configuration reload is triggered by sending a SIGHUP
to the Prometheus process or
sending a HTTP POST request to the /-/reload
endpoint.
This will also reload any configured rule files.
To specify which configuration file to load, use the -config.file
flag.
The file is written in YAML format, defined by the scheme described below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.
Generic placeholders are defined as follows:
<boolean>
: a boolean that can take the values true
or false
<duration>
: a duration matching the regular expression [0-9]+(ms|[smhdwy])
<labelname>
: a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
<labelvalue>
: a string of unicode characters<filename>
: a valid path in the current working directory<host>
: a valid string consisting of a hostname or IP followed by an optional port number<path>
: a valid URL path<scheme>
: a string that can take the values http
or https
<string>
: a regular string<secret>
: a regular string that is a secret, such as a passwordThe other placeholders are specified separately.
A valid example file can be found here.
The global configuration specifies parameters that are valid in all other configuration contexts. They also serve as defaults for other configuration sections.
global:
# How frequently to scrape targets by default.
[ scrape_interval: <duration> | default = 1m ]
# How long until a scrape request times out.
[ scrape_timeout: <duration> | default = 10s ]
# How frequently to evaluate rules.
[ evaluation_interval: <duration> | default = 1m ]
# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
[ <labelname>: <labelvalue> ... ]
# Rule files specifies a list of globs. Rules and alerts are read from
# all matching files.
rule_files:
[ - <filepath_glob> ... ]
# A list of scrape configurations.
scrape_configs:
[ - <scrape_config> ... ]
# Alerting specifies settings related to the Alertmanager.
alerting:
alert_relabel_configs:
[ - <relabel_config> ... ]
alertmanagers:
[ - <alertmanager_config> ... ]
# Settings related to the experimental remote write feature.
remote_write:
[ - <remote_write> ... ]
# Settings related to the experimental remote read feature.
remote_read:
[ - <remote_read> ... ]
<scrape_config>
A scrape_config
section specifies a set of targets and parameters describing how
to scrape them. In the general case, one scrape configuration specifies a single
job. In advanced configurations, this may change.
Targets may be statically configured via the static_configs
parameter or
dynamically discovered using one of the supported service-discovery mechanisms.
Additionally, relabel_configs
allow advanced modifications to any
target and its labels before scraping.
# The job name assigned to scraped metrics by default.
job_name: <job_name>
# How frequently to scrape targets from this job.
[ scrape_interval: <duration> | default = <global_config.scrape_interval> ]
# Per-scrape timeout when scraping this job.
[ scrape_timeout: <duration> | default = <global_config.scrape_timeout> ]
# The HTTP resource path on which to fetch metrics from targets.
[ metrics_path: <path> | default = /metrics ]
# honor_labels controls how Prometheus handles conflicts between labels that are
# already present in scraped data and labels that Prometheus would attach
# server-side ("job" and "instance" labels, manually configured target
# labels, and labels generated by service discovery implementations).
#
# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels. This is useful for use cases such as federation, where all labels
# specified in the target should be preserved.
#
# Note that any globally configured "external_labels" are unaffected by this
# setting. In communication with external systems, they are always applied only
# when a time series does not have a given label yet and are ignored otherwise.
[ honor_labels: <boolean> | default = false ]
# Configures the protocol scheme used for requests.
[ scheme: <scheme> | default = http ]
# Optional HTTP URL parameters.
params:
[ <string>: [<string>, ...] ]
# Sets the `Authorization` header on every scrape request with the
# configured username and password.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
# Sets the `Authorization` header on every scrape request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <secret> ]
# Sets the `Authorization` header on every scrape request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the scrape request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
# List of Azure service discovery configurations.
azure_sd_configs:
[ - <azure_sd_config> ... ]
# List of Consul service discovery configurations.
consul_sd_configs:
[ - <consul_sd_config> ... ]
# List of DNS service discovery configurations.
dns_sd_configs:
[ - <dns_sd_config> ... ]
# List of EC2 service discovery configurations.
ec2_sd_configs:
[ - <ec2_sd_config> ... ]
# List of OpenStack service discovery configurations.
openstack_sd_configs:
[ - <openstack_sd_config> ... ]
# List of file service discovery configurations.
file_sd_configs:
[ - <file_sd_config> ... ]
# List of GCE service discovery configurations.
gce_sd_configs:
[ - <gce_sd_config> ... ]
# List of Kubernetes service discovery configurations.
kubernetes_sd_configs:
[ - <kubernetes_sd_config> ... ]
# List of Marathon service discovery configurations.
marathon_sd_configs:
[ - <marathon_sd_config> ... ]
# List of AirBnB's Nerve service discovery configurations.
nerve_sd_configs:
[ - <nerve_sd_config> ... ]
# List of Zookeeper Serverset service discovery configurations.
serverset_sd_configs:
[ - <serverset_sd_config> ... ]
# List of Triton service discovery configurations.
triton_sd_configs:
[ - <triton_sd_config> ... ]
# List of labeled statically configured targets for this job.
static_configs:
[ - <static_config> ... ]
# List of target relabel configurations.
relabel_configs:
[ - <relabel_config> ... ]
# List of metric relabel configurations.
metric_relabel_configs:
[ - <relabel_config> ... ]
# Per-scrape limit on number of scraped samples that will be accepted.
# If more than this number of samples are present after metric relabelling
# the entire scrape will be treated as failed. 0 means no limit.
[ sample_limit: <int> | default = 0 ]
Where <job_name>
must be unique across all scrape configurations.
<tls_config>
A tls_config
allows configuring TLS connections.
# CA certificate to validate API server certificate with.
[ ca_file: <filename> ]
# Certificate and key files for client cert authentication to the server.
[ cert_file: <filename> ]
[ key_file: <filename> ]
# ServerName extension to indicate the name of the server.
# http://tools.ietf.org/html/rfc4366#section-3.1
[ server_name: <string> ]
# Disable validation of the server certificate.
[ insecure_skip_verify: <boolean> ]
<azure_sd_config>
Azure SD configurations allow retrieving scrape targets from Azure VMs.
The following meta labels are available on targets during relabeling:
__meta_azure_machine_id
: the machine ID__meta_azure_machine_location
: the location the machine runs in__meta_azure_machine_name
: the machine name__meta_azure_machine_private_ip
: the machine's private IP__meta_azure_machine_resource_group
: the machine's resource group__meta_azure_machine_tag_<tagname>
: each tag value of the machineSee below for the configuration options for Azure discovery:
# The information to access the Azure API.
# The subscription ID.
subscription_id: <string>
# The tenant ID.
tenant_id: <string>
# The client ID.
client_id: <string>
# The client secret.
client_secret: <secret>
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 300s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
<consul_sd_config>
Consul SD configurations allow retrieving scrape targets from Consul's Catalog API.
The following meta labels are available on targets during relabeling:
__meta_consul_address
: the address of the target__meta_consul_dc
: the datacenter name for the target__meta_consul_metadata_<key>
: each metadata key value of the target__meta_consul_node
: the node name defined for the target__meta_consul_service_address
: the service address of the target__meta_consul_service_id
: the service ID of the target__meta_consul_service_port
: the service port of the target__meta_consul_service
: the name of the service the target belongs to__meta_consul_tags
: the list of tags of the target joined by the tag separator# The information to access the Consul API. It is to be defined
# as the Consul documentation requires.
server: <host>
[ token: <secret> ]
[ datacenter: <string> ]
[ scheme: <string> ]
[ username: <string> ]
[ password: <secret> ]
# A list of services for which targets are retrieved. If omitted, all services
# are scraped.
services:
[ - <string> ]
# The string by which Consul tags are joined into the tag label.
[ tag_separator: <string> | default = , ]
Note that the IP number and port used to scrape the targets is assembled as
<__meta_consul_address>:<__meta_consul_service_port>
. However, in some
Consul setups, the relevant address is in __meta_consul_service_address
.
In those cases, you can use the relabel
feature to replace the special __address__
label.
<dns_sd_config>
A DNS-based service discovery configuration allows specifying a set of DNS
domain names which are periodically queried to discover a list of targets. The
DNS servers to be contacted are read from /etc/resolv.conf
.
This service discovery method only supports basic DNS A, AAAA and SRV record queries, but not the advanced DNS-SD approach specified in RFC6763.
During the relabeling phase, the meta label
__meta_dns_name
is available on each target and is set to the
record name that produced the discovered target.
# A list of DNS domain names to be queried.
names:
[ - <domain_name> ]
# The type of DNS query to perform.
[ type: <query_type> | default = 'SRV' ]
# The port number used if the query type is not SRV.
[ port: <number>]
# The time after which the provided names are refreshed.
[ refresh_interval: <duration> | default = 30s ]
Where <domain_name>
is a valid DNS domain name.
Where <query_type>
is SRV
, A
, or AAAA
.
<ec2_sd_config>
EC2 SD configurations allow retrieving scrape targets from AWS EC2 instances. The private IP address is used by default, but may be changed to the public IP address with relabeling.
The following meta labels are available on targets during relabeling:
__meta_ec2_availability_zone
: the availability zone in which the instance is running__meta_ec2_instance_id
: the EC2 instance ID__meta_ec2_instance_state
: the state of the EC2 instance__meta_ec2_instance_type
: the type of the EC2 instance__meta_ec2_private_ip
: the private IP address of the instance, if present__meta_ec2_public_dns_name
: the public DNS name of the instance, if available__meta_ec2_public_ip
: the public IP address of the instance, if available__meta_ec2_subnet_id
: comma separated list of subnets IDs in which the instance is running, if available__meta_ec2_tag_<tagkey>
: each tag value of the instance__meta_ec2_vpc_id
: the ID of the VPC in which the instance is running, if availableSee below for the configuration options for EC2 discovery:
# The information to access the EC2 API.
# The AWS Region.
region: <string>
# The AWS API keys. If blank, the environment variables `AWS_ACCESS_KEY_ID`
# and `AWS_SECRET_ACCESS_KEY` are used.
[ access_key: <string> ]
[ secret_key: <secret> ]
# Named AWS profile used to connect to the API.
[ profile: <string> ]
# AWS Role ARN, an alternative to using AWS API keys.
[ role_arn: <string> ]
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
<openstack_sd_config>
OpenStack SD configurations allow retrieving scrape targets from OpenStack Nova instances.
The following meta labels are available on targets during relabeling:
__meta_openstack_instance_id
: the OpenStack instance ID__meta_openstack_instance_name
: the OpenStack instance name__meta_openstack_instance_status
: the status of the OpenStack instance__meta_openstack_instance_flavor
: the flavor of the OpenStack instance__meta_openstack_public_ip
: the public IP of the OpenStack instance__meta_openstack_private_ip
: the private IP of the OpenStack instance__meta_openstack_tag_<tagkey>
: each tag value of the instanceSee below for the configuration options for OpenStack discovery:
# The information to access the OpenStack API.
# The OpenStack Region.
region: <string>
# identity_endpoint specifies the HTTP endpoint that is required to work with
# the Identity API of the appropriate version. While it's ultimately needed by
# all of the identity services, it will often be populated by a provider-level
# function.
[ identity_endpoint: <string> ]
# username is required if using Identity V2 API. Consult with your provider's
# control panel to discover your account's username. In Identity V3, either
# userid or a combination of username and domain_id or domain_name are needed.
[ username: <string> ]
[ userid: <string> ]
# password for the Identity V2 and V3 APIs. Consult with your provider's
# control panel to discover your account's preferred method of authentication.
[ password: <secret> ]
# At most one of domain_id and domain_name must be provided if using username
# with Identity V3. Otherwise, either are optional.
[ domain_name: <string> ]
[ domain_id: <string> ]
# The project_id and project_name fields are optional for the Identity V2 API.
# Some providers allow you to specify a project_name instead of the project_id.
# Some require both. Your provider's authentication policies will determine
# how these fields influence authentication.
[ project_name: <string> ]
[ project_id: <string> ]
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
<file_sd_config>
File-based service discovery provides a more generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms.
It reads a set of files containing a list of zero or more
<static_config>
s. Changes to all defined files are detected via disk watches
and applied immediately. Files may be provided in YAML or JSON format. Only
changes resulting in well-formed target groups are applied.
The JSON file must contain a list of static configs, using this format:
[
{
"targets": [ "<host>", ... ],
"labels": {
"<labelname>": "<labelvalue>", ...
}
},
...
]
As a fallback, the file contents are also re-read periodically at the specified refresh interval.
Each target has a meta label __meta_filepath
during the
relabeling phase. Its value is set to the
filepath from which the target was extracted.
There is a list of integrations with this discovery mechanism.
# Patterns for files from which target groups are extracted.
files:
[ - <filename_pattern> ... ]
# Refresh interval to re-read the files.
[ refresh_interval: <duration> | default = 5m ]
Where <filename_pattern>
may be a path ending in .json
, .yml
or .yaml
. The last path segment
may contain a single *
that matches any character sequence, e.g. my/path/tg_*.json
.
<gce_sd_config>
GCE SD configurations allow retrieving scrape targets from GCP GCE instances. The private IP address is used by default, but may be changed to the public IP address with relabeling.
The following meta labels are available on targets during relabeling:
__meta_gce_instance_name
: the name of the instance__meta_gce_metadata_<name>
: each metadata item of the instance__meta_gce_network
: the network URL of the instance__meta_gce_private_ip
: the private IP address of the instance__meta_gce_project
: the GCP project in which the instance is running__meta_gce_public_ip
: the public IP address of the instance, if present__meta_gce_subnetwork
: the subnetwork URL of the instance__meta_gce_tags
: comma separated list of instance tags__meta_gce_zone
: the GCE zone URL in which the instance is runningSee below for the configuration options for GCE discovery:
# The information to access the GCE API.
# The GCP Project
project: <string>
# The zone of the scrape targets. If you need multiple zones use multiple
# gce_sd_configs.
zone: <string>
# Filter can be used optionally to filter the instance list by other criteria
# Syntax of this filter string is described here in the filter query parameter section:
# https://cloud.google.com/compute/docs/reference/latest/instances/list
[ filter: <string> ]
# Refresh interval to re-read the instance list
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
# The tag separator is used to separate the tags on concatenation
[ tag_separator: <string> | default = , ]
Credentials are discovered by the Google Cloud SDK default client by looking in the following places, preferring the first location found:
GOOGLE_APPLICATION_CREDENTIALS
environment variable$HOME/.config/gcloud/application_default_credentials.json
If Prometheus is running within GCE, the service account associated with the instance it is running on should have at least read-only permissions to the compute resources. If running outside of GCE make sure to create an appropriate service account and place the credential file in one of the expected locations.
<kubernetes_sd_config>
Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API and always staying synchronized with the cluster state.
One of the following role
types can be configured to discover targets:
node
The node
role discovers one target per cluster node with the address defaulting
to the Kubelet's HTTP port.
The target address defaults to the first existing address of the Kubernetes
node object in the address type order of NodeInternalIP
, NodeExternalIP
,
NodeLegacyHostIP
, and NodeHostName
.
Available meta labels:
__meta_kubernetes_node_name
: The name of the node object.__meta_kubernetes_node_label_<labelname>
: Each label from the node object.__meta_kubernetes_node_annotation_<annotationname>
: Each annotation from the node object.__meta_kubernetes_node_address_<address_type>
: The first address for each node address type, if it exists.In addition, the instance
label for the node will be set to the node name
as retrieved from the API server.
service
The service
role discovers a target for each service port for each service.
This is generally useful for blackbox monitoring of a service.
The address will be set to the Kubernetes DNS name of the service and respective
service port.
Available meta labels:
__meta_kubernetes_namespace
: The namespace of the service object.__meta_kubernetes_service_name
: The name of the service object.__meta_kubernetes_service_label_<labelname>
: The label of the service object.__meta_kubernetes_service_annotation_<annotationname>
: The annotation of the service object.__meta_kubernetes_service_port_name
: Name of the service port for the target.__meta_kubernetes_service_port_number
: Number of the service port for the target.__meta_kubernetes_service_port_protocol
: Protocol of the service port for the target.pod
The pod
role discovers all pods and exposes their containers as targets. For each declared
port of a container, a single target is generated. If a container has no specified ports,
a port-free target per container is created for manually adding a port via relabeling.
Available meta labels:
__meta_kubernetes_namespace
: The namespace of the pod object.__meta_kubernetes_pod_name
: The name of the pod object.__meta_kubernetes_pod_ip
: The pod IP of the pod object.__meta_kubernetes_pod_label_<labelname>
: The label of the pod object.__meta_kubernetes_pod_annotation_<annotationname>
: The annotation of the pod object.__meta_kubernetes_pod_container_name
: Name of the container the target address points to.__meta_kubernetes_pod_container_port_name
: Name of the container port.__meta_kubernetes_pod_container_port_number
: Number of the container port.__meta_kubernetes_pod_container_port_protocol
: Protocol of the container port.__meta_kubernetes_pod_ready
: Set to true
or false
for the pod's ready state.__meta_kubernetes_pod_node_name
: The name of the node the pod is scheduled onto.__meta_kubernetes_pod_host_ip
: The current host IP of the pod object.endpoints
The endpoints
role discovers targets from listed endpoints of a service. For each endpoint
address one target is discovered per port. If the endpoint is backed by a pod, all
additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
Available meta labels:
__meta_kubernetes_namespace
: The namespace of the endpoints object.__meta_kubernetes_endpoints_name
: The names of the endpoints object.__meta_kubernetes_endpoint_ready
: Set to true
or false
for the endpoint's ready state.__meta_kubernetes_endpoint_port_name
: Name of the endpoint port.__meta_kubernetes_endpoint_port_protocol
: Protocol of the endpoint port.role: service
discovery are attached.role: pod
discovery are attached.ingress
The ingress
role discovers a target for each path of each ingress.
This is generally useful for blackbox monitoring of an ingress.
The address will be set to the host specified in the ingress spec.
Available meta labels:
__meta_kubernetes_namespace
: The namespace of the ingress object.__meta_kubernetes_ingress_name
: The name of the ingress object.__meta_kubernetes_ingress_label_<labelname>
: The label of the ingress object.__meta_kubernetes_ingress_annotation_<annotationname>
: The annotation of the ingress object.__meta_kubernetes_ingress_scheme
: Protocol scheme of ingress, https
if TLS
config is set. Defaults to http
.__meta_kubernetes_ingress_path
: Path from ingress spec. Defaults to /
.See below for the configuration options for Kubernetes discovery:
# The information to access the Kubernetes API.
# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]
# The Kubernetes role of entities that should be discovered.
role: <role>
# Optional authentication information used to authenticate to the API server.
# Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
# mutually exclusive.
# Optional HTTP basic authentication information.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
# Optional bearer token authentication information.
[ bearer_token: <secret> ]
# Optional bearer token file authentication information.
[ bearer_token_file: <filename> ]
# TLS configuration.
tls_config:
[ <tls_config> ]
# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
names:
[ - <string> ]
Where <role>
must be endpoints
, service
, pod
, or node
.
See this example Prometheus configuration file for a detailed example of configuring Prometheus for Kubernetes.
You may wish to check out the 3rd party Prometheus Operator, which automates the Prometheus setup on top of Kubernetes.
<marathon_sd_config>
Marathon SD configurations allow retrieving scrape targets using the Marathon REST API. Prometheus will periodically check the REST endpoint for currently running tasks and create a target group for every app that has at least one healthy task.
The following meta labels are available on targets during relabeling:
__meta_marathon_app
: the name of the app (with slashes replaced by dashes)__meta_marathon_image
: the name of the Docker image used (if available)__meta_marathon_task
: the ID of the Mesos task__meta_marathon_app_label_<labelname>
: any Marathon labels attached to the app__meta_marathon_port_definition_label_<labelname>
: the port definition labels__meta_marathon_port_mapping_label_<labelname>
: the port mapping labels__meta_marathon_port_index
: the port index number (e.g. 1
for PORT1
)See below for the configuration options for Marathon discovery:
# List of URLs to be used to contact Marathon servers.
# You need to provide at least one server URL, but should provide URLs for
# all masters you have running.
servers:
- <string>
# Optional bearer token authentication information.
# It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <secret> ]
# Optional bearer token file authentication information.
# It is mutually exclusive with `bearer_token`.
[ bearer_token_file: <filename> ]
# Polling interval
[ refresh_interval: <duration> | default = 30s ]
By default every app listed in Marathon will be scraped by Prometheus. If not all of your services provide Prometheus metrics, you can use a Marathon label and Prometheus relabeling to control which instances will actually be scraped. Also by default all apps will show up as a single job in Prometheus (the one specified in the configuration file), which can also be changed using relabeling.
<nerve_sd_config>
Nerve SD configurations allow retrieving scrape targets from AirBnB's Nerve which are stored in Zookeeper.
The following meta labels are available on targets during relabeling:
__meta_nerve_path
: the full path to the endpoint node in Zookeeper__meta_nerve_endpoint_host
: the host of the endpoint__meta_nerve_endpoint_port
: the port of the endpoint__meta_nerve_endpoint_name
: the name of the endpoint# The Zookeeper servers.
servers:
- <host>
# Paths can point to a single service, or the root of a tree of services.
paths:
- <string>
[ timeout: <duration> | default = 10s ]
<serverset_sd_config>
Serverset SD configurations allow retrieving scrape targets from Serversets which are stored in Zookeeper. Serversets are commonly used by Finagle and Aurora.
The following meta labels are available on targets during relabeling:
__meta_serverset_path
: the full path to the serverset member node in Zookeeper__meta_serverset_endpoint_host
: the host of the default endpoint__meta_serverset_endpoint_port
: the port of the default endpoint__meta_serverset_endpoint_host_<endpoint>
: the host of the given endpoint__meta_serverset_endpoint_port_<endpoint>
: the port of the given endpoint__meta_serverset_shard
: the shard number of the member__meta_serverset_status
: the status of the member# The Zookeeper servers.
servers:
- <host>
# Paths can point to a single serverset, or the root of a tree of serversets.
paths:
- <string>
[ timeout: <duration> | default = 10s ]
Serverset data must be in the JSON format, the Thrift format is not currently supported.
<triton_sd_config>
Triton SD configurations allow retrieving scrape targets from Container Monitor discovery endpoints.
The following meta labels are available on targets during relabeling:
__meta_triton_machine_id
: the UUID of the target container__meta_triton_machine_alias
: the alias of the target container__meta_triton_machine_image
: the target containers image type__meta_triton_machine_server_id
: the server UUID for the target container# The information to access the Triton discovery API.
# The account to use for discovering new target containers.
account: <string>
# The DNS suffix which should be applied to target containers.
dns_suffix: <string>
# The Triton discovery endpoint (e.g. 'cmon.us-east-3b.triton.zone'). This is
# often the same value as dns_suffix.
endpoint: <string>
# The port to use for discovery and metric scraping.
[ port: <int> | default = 9163 ]
# The interval which should should be used for refreshing target containers.
[ refresh_interval: <duration> | default = 60s ]
# The Triton discovery API version.
[ version: <int> | default = 1 ]
# TLS configuration.
tls_config:
[ <tls_config> ]
<static_config>
A static_config
allows specifying a list of targets and a common label set
for them. It is the canonical way to specify static targets in a scrape
configuration.
# The targets specified by the static config.
targets:
[ - '<host>' ]
# Labels assigned to all metrics scraped from the targets.
labels:
[ <labelname>: <labelvalue> ... ]
<relabel_config>
Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped. Multiple relabeling steps can be configured per scrape configuration. They are applied to the label set of each target in order of their appearance in the configuration file.
Initially, aside from the configured per-target labels, a target's job
label is set to the job_name
value of the respective scrape configuration.
The __address__
label is set to the <host>:<port>
address of the target.
After relabeling, the instance
label is set to the value of __address__
by default if
it was not set during relabeling. The __scheme__
and __metrics_path__
labels
are set to the scheme and metrics path of the target respectively. The __param_<name>
label is set to the value of the first passed URL parameter called <name>
.
Additional labels prefixed with __meta_
may be available during the
relabeling phase. They are set by the service discovery mechanism that provided
the target and vary between mechanisms.
Labels starting with __
will be removed from the label set after relabeling is completed.
If a relabeling step needs to store a label value only temporarily (as the
input to a subsequent relabeling step), use the __tmp
label name prefix. This
prefix is guaranteed to never be used by Prometheus itself.
# The source labels select values from existing labels. Their content is concatenated
# using the configured separator and matched against the configured regular expression
# for the replace, keep, and drop actions.
[ source_labels: '[' <labelname> [, ...] ']' ]
# Separator placed between concatenated source label values.
[ separator: <string> | default = ; ]
# Label to which the resulting value is written in a replace action.
# It is mandatory for replace actions. Regex capture groups are available.
[ target_label: <labelname> ]
# Regular expression against which the extracted value is matched.
[ regex: <regex> | default = (.*) ]
# Modulus to take of the hash of the source label values.
[ modulus: <uint64> ]
# Replacement value against which a regex replace is performed if the
# regular expression matches. Regex capture groups are available.
[ replacement: <string> | default = $1 ]
# Action to perform based on regex matching.
[ action: <relabel_action> | default = replace ]
<regex>
is any valid
RE2 regular expression. It is
required for the replace
, keep
, drop
, labelmap
,labeldrop
and labelkeep
actions. The regex is
anchored on both ends. To un-anchor the regex, use .*<regex>.*
.
<relabel_action>
determines the relabeling action to take:
replace
: Match regex
against the concatenated source_labels
. Then, set
target_label
to replacement
, with match group references
(${1}
, ${2}
, ...) in replacement
substituted by their value. If regex
does not match, no replacement takes place.keep
: Drop targets for which regex
does not match the concatenated source_labels
.drop
: Drop targets for which regex
matches the concatenated source_labels
.hashmod
: Set target_label
to the modulus
of a hash of the concatenated source_labels
.labelmap
: Match regex
against all label names. Then copy the values of the matching labels
to label names given by replacement
with match group references
(${1}
, ${2}
, ...) in replacement
substituted by their value.labeldrop
: Match regex
against all label names. Any label that matches will be
removed from the set of labels.labelkeep
: Match regex
against all label names. Any label that does not match will be
removed from the set of labels.Care must be taken with labeldrop
and labelkeep
to ensure that metrics are still uniquely labeled
once the labels are removed.
<metric_relabel_configs>
Metric relabeling is applied to samples as the last step before ingestion. It
has the same configuration format and actions as target relabeling. Metric
relabeling does not apply to automatically generated timeseries such as up
.
One use for this is to blacklist time series that are too expensive to ingest.
<alert_relabel_configs>
Alert relabeling is applied to alerts before they are sent to the Alertmanager. It has the same configuration format and actions as target relabeling. Alert relabeling is applied after external labels.
One use for this is ensuring a HA pair of Prometheus servers with different external labels send identical alerts.
<alertmanager_config>
-alertmanager.url
flag
as a stable alternative.An alertmanager_config
section specifies Alertmanager instances the Prometheus server sends
alerts to. It also provides parameters to configure how to communicate with these Alertmanagers.
Alertmanagers may be statically configured via the static_configs
parameter or
dynamically discovered using one of the supported service-discovery mechanisms.
Additionally, relabel_configs
allow selecting Alertmanagers from discovered
entities and provide advanced modifications to the used API path, which is exposed
through the __alerts_path__
label.
# Per-target Alertmanager timeout when pushing alerts.
[ timeout: <duration> | default = 10s ]
# Prefix for the HTTP path alerts are pushed to.
[ path_prefix: <path> | default = / ]
# Configures the protocol scheme used for requests.
[ scheme: <scheme> | default = http ]
# Sets the `Authorization` header on every request with the
# configured username and password.
basic_auth:
[ username: <string> ]
[ password: <string> ]
# Sets the `Authorization` header on every request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the scrape request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
# List of Azure service discovery configurations.
azure_sd_configs:
[ - <azure_sd_config> ... ]
# List of Consul service discovery configurations.
consul_sd_configs:
[ - <consul_sd_config> ... ]
# List of DNS service discovery configurations.
dns_sd_configs:
[ - <dns_sd_config> ... ]
# List of EC2 service discovery configurations.
ec2_sd_configs:
[ - <ec2_sd_config> ... ]
# List of file service discovery configurations.
file_sd_configs:
[ - <file_sd_config> ... ]
# List of GCE service discovery configurations.
gce_sd_configs:
[ - <gce_sd_config> ... ]
# List of Kubernetes service discovery configurations.
kubernetes_sd_configs:
[ - <kubernetes_sd_config> ... ]
# List of Marathon service discovery configurations.
marathon_sd_configs:
[ - <marathon_sd_config> ... ]
# List of AirBnB's Nerve service discovery configurations.
nerve_sd_configs:
[ - <nerve_sd_config> ... ]
# List of Zookeeper Serverset service discovery configurations.
serverset_sd_configs:
[ - <serverset_sd_config> ... ]
# List of Triton service discovery configurations.
triton_sd_configs:
[ - <triton_sd_config> ... ]
# List of labeled statically configured Alertmanagers.
static_configs:
[ - <static_config> ... ]
# List of Alertmanager relabel configurations.
relabel_configs:
[ - <relabel_config> ... ]
<remote_write>
write_relabel_configs
is relabeling applied to samples before sending them
to the remote endpoint. Write relabeling is applied after external labels. This
could be used to limit which samples are sent.
There is a small demo of how to use this functionality.
# The URL of the endpoint to send samples to.
url: <string>
# Timeout for requests to the remote write endpoint.
[ remote_timeout: <duration> | default = 30s ]
# List of remote write relabel configurations.
write_relabel_configs:
[ - <relabel_config> ... ]
# Sets the `Authorization` header on every remote write request with the
# configured username and password.
basic_auth:
[ username: <string> ]
[ password: <string> ]
# Sets the `Authorization` header on every remote write request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every remote write request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the remote write request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
There is a list of integrations with this feature.
<remote_read>
# The URL of the endpoint to query from.
url: <string>
# Timeout for requests to the remote read endpoint.
[ remote_timeout: <duration> | default = 30s ]
# Sets the `Authorization` header on every remote read request with the
# configured username and password.
basic_auth:
[ username: <string> ]
[ password: <string> ]
# Sets the `Authorization` header on every remote read request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every remote read request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the remote read request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
There is a list of integrations with this feature.
This documentation is open-source. Please help improve it by filing issues or pull requests.