Pushing metrics

Occasionally you will need to monitor components which cannot be scraped. The Prometheus Pushgateway allows you to push time series from short-lived service-level batch jobs to an intermediary job which Prometheus can scrape. Combined with Prometheus's simple text-based exposition format, this makes it easy to instrument even shell scripts without a client library.

Java batch job example

This example illustrates how to instrument a batch job and alert on it not having succeeded recently.

If using Maven, add the following to pom.xml:


Instrument your batch job's code:

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.Gauge;
import io.prometheus.client.exporter.PushGateway;

void executeBatchJob() throws Exception {
 CollectorRegistry registry = new CollectorRegistry();
 Gauge duration = Gauge.build()
     .help("Duration of my batch job in seconds.")
 Gauge.Timer durationTimer = duration.startTimer();
 try {
   // Your code here.

   // This is only added to the registry after success,
   // so that a previous success in the Pushgateway is not overwritten on failure.
   Gauge lastSuccess = Gauge.build()
       .help("Last time my batch job succeeded, in unixtime.")
 } finally {
   PushGateway pg = new PushGateway("");
   pg.pushAdd(registry, "my_batch_job");

Set up a Pushgateway and update the host and port in the above code if needed.

Set up an alert to fire if the job has not run recently. Add the following to the rules of a Prometheus server that is scraping the Pushgateway:

- name: MyBatchJob
  - alert: MyBatchJobNotCompleted
    expr: min(time() - my_batch_job_last_success_unixtime{job="my_batch_job"}) > 60 * 60
    for: 5m
      severity: page
      summary: MyBatchJob has not completed successfully in over an hour

This documentation is open-source. Please help improve it by filing issues or pull requests.