How to create a Prometheus recording rule

A Prometheus recording rule stores the result of a PromQL expression as a new time series. Create one when a dashboard, alert, or report repeatedly needs the same aggregate and the source query should not be recomputed every time it is used.

Prometheus loads recording rules from YAML rule files listed under rule_files in /etc/prometheus/prometheus.yml or the active configuration file passed with --config.file. Each rule belongs to a group, and the group interval controls how often Prometheus evaluates the expression.

The sample rule aggregates Prometheus's own HTTP request metric, and the final API query checks the recorded series after the server reloads the rule file. Replace the expression and record name with the service metric that should feed your own dashboard or alert.

Steps to create a Prometheus recording rule:

Create a directory for Prometheus rule files.

$ sudo install -d -m 0755 /etc/prometheus/rules

Open a new recording rule file.

$ sudoedit /etc/prometheus/rules/recording-rules.yml

Add a rule group with one recording rule.

groups:
  - name: prometheus-http-rates
    interval: 30s
    rules:
      - record: job:prometheus_http_requests:rate5m
        expr: sum by (job) (rate(prometheus_http_requests_total[5m]))
        labels:
          rule_type: recording

The record value becomes the new metric name. Use a valid metric name and an expression that returns the label set the dashboard or alert should query.

Open the active Prometheus configuration file.
```
$ sudoedit /etc/prometheus/prometheus.yml
```
Add the rule file path under rule_files.
```
global:
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/rules/*.yml
```
Keep existing scrape_configs and other settings in the file. The rule_files paths can point to one file or to a glob that matches several rule files.

Related: How to load a Prometheus rule file

Check the recording rule syntax.

$ promtool check rules /etc/prometheus/rules/recording-rules.yml
Checking /etc/prometheus/rules/recording-rules.yml
  SUCCESS: 1 rules found

Check the full Prometheus configuration before reloading.

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Checking /etc/prometheus/rules/recording-rules.yml
  SUCCESS: 1 rules found

Reload Prometheus so it reads the new rule file.
```
$ curl -sS -X POST http://localhost:9090/-/reload
```
The HTTP reload endpoint requires Prometheus to run with --web.enable-lifecycle. If that flag is not enabled, reload with SIGHUP or the service manager used on the host.

Related: How to reload Prometheus configuration
Wait for the next rule evaluation interval.

With interval: 30s, wait at least 30 seconds after the reload before checking the recorded metric. A rate expression can return an empty result until the source metric has enough scraped samples in the range.

Query the recorded metric from Prometheus.

$ curl -sS 'http://localhost:9090/api/v1/query?query=job:prometheus_http_requests:rate5m'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"job:prometheus_http_requests:rate5m","job":"prometheus","rule_type":"recording"},"value":[1781949879.580,"0.08811999999999999"]}]}}

The status value should be success, and the result should include the recording rule metric name. An empty result usually means Prometheus has loaded the rule but has not evaluated a sample yet.

Author: Mohd Shakir Zakaria
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.