Go Back

Metrics

Prometheus

Prometheus is really good at one thing - collecting metrics and storing them for short-term analysis. It essentially consists of an executable and one or more config files.

The configuration is passed to the prometheus executable using the --config.file parameter like this:

prometheus --config.file prometheus.yml
  

The config file can be quite simple, specifying a target to collect metrics from:

scrape_configs:
  - job_name: "myself"
    static_configs:
      - targets: ["localhost:9090"]
        labels:
          app: "prometheus"
  

All we did is provide the host name and port for Prometheus to collect metrics from itself, and Prometheus uses the default path of "/metrics" (override by setting metrics_path) to read the metrics from. It defaults to the HTTP scheme (i.e. no TLS). Prometheus provides us with demo targets at "demo.promlabs.com:10000", "demo.promlabs.com:10001", ...

A look at the sample data sources shows you how you can expose metrics to Prometheus with what is essentially text content. Metrics can be of the type "gauge", "counter", or "histogram" (see: https://prometheus.io/docs/concepts/metric_types/).

# HELP pockettheories_temperature_sensor The temperature from the room sensor
# TYPE pockettheories_temperature_sensor gauge
pockettheories_temperature_sensor{building="A",room="401"} 25

# HELP pockettheories_request_counter The number of web requests at the endpoint
# TYPE pockettheories_request_counter counter
pockettheories_request_counter{method="GET",path="/index.html"} 25
    

The following is some simple Ruby code to expose metrics for Prometheus

nitin@MacBook prometheus % cat Gemfile
source :rubygems
gem 'sinatra'
gem 'rackup'
gem 'puma'

nitin@MacBook prometheus % cat stats_interface.rb 
require 'sinatra'

get '/metrics' do
content_type 'text/plain'
'temperature_sensor{room="A1"} 25'
end
    

The web UI provided by Prometheus on the default port 9090 can be used to run PromQL queries.

Prometheus can return either an instant vector (the default, latest values) or a range vector (when using square brackets with a time window, values within the specified look-behind window).

Prometheus also provides functions such as rate() and delta(). These apply to specific data types (counter, gauge, histogram). Look up the documentation to see what kind of input the function takes.

Here are some PromQL queries.

temperature_sensor{app="my_pretend_iot"}
temperature_sensor{app="my_pretend_iot"}[5m]
delta(temperature_sensor{app="my_pretend_iot"}[5m])
rate(web_hit_count{method="GET",path="/index.html"})
    

Static target configuration is rather uncommon because it becomes difficult to manage with a huge number of targets.

Here's a config file that uses a service discovery file:

scrape_configs:
  - job_name: "myself"
    file_sd_configs:
      - files:
        - "targets.json"
        refresh_interval: 60s
    relabel_configs:
      - source_labels: [__meta_ec2_instance_id]
        target_label: ec2_instance
  

With my targets.json being the following:

[{
    "targets": ["127.0.0.1:9090"],
    "labels": {
        "__metrics_path__": "/more_metrics"
    }
}]
  

Prometheus can summarize the data using the sum(), avg(), max(), min() functions.

avg(rate(web_hit_count[5m]))
sum by(path) rate(web_hit_count[5m]))
sum(sum by(path) rate(web_hit_count[5m])))
max_over_time(temperature_sensor[15m])
    

Also see: histogram_quantile()