10.2. Cache PromQL query guide

The PromQL query syntax supported in custom dashboard and report panels.

Overview

Cache PromQL is the Prometheus-compatible query language used to describe the metric a panel shows in custom dashboards and in the custom reports of the SRE report. It supports the widely used subset of standard PromQL and evaluates against the metric cache the product has already computed.

You write these queries in three places.

• The PromQL query of a custom dashboard panel
• The PromQL panel of a custom report
• The query type of a variable (label_values(...))

This document is a reference for people building panels themselves. It sets out which operators and functions are available, what is not supported, and the traps people commonly fall into.

Note: you do not need to know PromQL to build most screens — start from a template in custom dashboards or the SRE report. Come here when you modify a template or assemble metrics yourself.

How it works

Cache PromQL evaluates queries not against an ordinary Prometheus server but against the metrics the product has collected, aggregated and stored in its cache. This has a few practical consequences.

• Time series first: the result is a time series over a range. Single value (stat) and table panels use the series' most recent value.
• Cache resolution: values follow the resolution of the interval (step) stored in the cache, not raw samples.

Metric Explorer — finding out which metrics exist

To write a query you first need to know which metrics exist. You do not have to memorise their names — explore the cache directly as follows.

Metric Explorer (at /api/v1/metrics-explorer) is a dedicated screen for exploring every metric in the cache. Search for and select a metric on the left and the right shows its labels (dimensions) with the number of values each has, the aggregation options, a time-series chart, and the individual series. It tells you at a glance which metrics exist and which labels you can narrow by, without memorising any names.

Listing the metric names

The cache carries Prometheus's standard meta label __name__ (the metric name) as it is. You can use it to pull out every available metric name as a list of values.

• Create a variable on a custom dashboard, set its type to query, and set the query to label_values(__name__).
• Save it and the variable combo in the header becomes a dropdown of every metric name in the cache. Scan it for the metric you want.
• To see only a particular prefix, type a search term into the combo (container_http, node_, kube_).

This variable is for exploration only; delete it once you have found the metric. For how to create variables, see the "Filtering with variables" section of custom dashboards.

Finding a metric's labels (dimensions)

Once you have chosen a metric, find out which labels it carries (namespace, pod, app_id and so on) so you can use them as matchers and aggregation keys.

• Through the preview: in the panel editor dialog of custom dashboards, enter just the metric name and press Preview; the legend of the resulting series shows the labels.
• Enumerating label values: label_values(<metric>, <label>) lists a label's values. For example label_values(container_http_requests_count, namespace) gives the namespaces the metric exists in.

A catalogue of the main metrics

The metrics in common use, by category. For the full list, use label_values(__name__) above.

Nodes (hosts) — node_*

Metric	Description
node_cpu_usage_percent · node_cpu_used · node_cpu_cores	CPU usage (%), cores used, total cores
node_memory_usage_bytes · node_memory_usage_percent · node_memory_available_bytes	Memory used, usage rate, available
node_disk_read_bytes · node_disk_written_bytes · node_disk_space_bytes · node_disk_io_time	Disk reads, writes, capacity, I/O time
node_net_rx_bytes · node_net_tx_bytes · node_net_rx_dropped · node_net_tx_dropped	Network bytes received and sent, and drops
node_load_average_1m · node_load_average_5m · node_load_average_15m	Load averages
node_gpu_utilization_percent_avg · node_gpu_memory_used_bytes · node_gpu_power_usage_watts · node_gpu_temperature_celsius	GPU utilization, memory, power, temperature
node_uptime_seconds · node_info	Uptime, node metadata

Container and application resources — container_*

Metric	Description
container_cpu_usage · container_cpu_limit · container_throttled_time	CPU used, limit, throttled time
container_memory_rss · container_memory_cache · container_memory_limit	Memory RSS, cache, limit
container_restarts · container_oom_kills_total	Restart count, OOM kills
container_net_tcp_bytes_sent · container_net_tcp_bytes_received · container_net_tcp_active_connections · container_net_latency	TCP bytes sent and received, active connections, latency
container_volume_used · container_volume_size	Volume used and total capacity
container_log_messages	Log message count (with a level label)
container_info · container_application_type	Container and application type metadata

L7 requests and queries (by protocol) — container_<protocol>_*

Application traffic instrumented with eBPF. Each protocol uses the same naming pattern.

Metric pattern	Description
container_http_requests_count	HTTP request rate (per second — the rate is already applied; do not wrap it)
container_http_requests_total	Cumulative HTTP requests
container_http_requests_histogram · ..._duration_seconds_total_bucket	The response time histogram (for percentiles)
container_http_requests_latency_total	Total request-seconds (for approximating average latency)
container_http_security_events_count · container_http_geo_country_pct	Security event count, and the geographic distribution of requests

Beyond HTTP, the database and messaging protocols follow the same pattern with either _requests_ or _queries_.

• Request-style (_requests_): container_kafka_requests_* · container_zookeeper_requests_*
• Query-style (_queries_): container_postgres_queries_* · container_mysql_queries_* · container_mongo_queries_* · container_oracle_queries_* · container_cassandra_queries_* · container_clickhouse_queries_* · container_memcached_queries_*
• Others: container_dns_requests_total · container_dns_requests_latency · container_nats_messages_total

The suffix convention: within a family, _count is a per-second rate (already applied), _total is a cumulative count, _histogram/_bucket are for percentiles, and _latency_total is total request-seconds. The rate is already applied to the _count family, so do not wrap it in rate() or increase() (see the caveat below).

Language runtimes — container_jvm_* · container_dotnet_* · container_python_*

Metric (examples)	Description
container_jvm_heap_used_bytes · container_jvm_heap_size_bytes · container_jvm_gc_time_seconds · container_jvm_threads_live	JVM heap, GC, threads
container_dotnet_memory_heap_size_bytes · container_dotnet_gc_count_total · container_dotnet_thread_pool_size	.NET heap, GC, thread pool
container_python_thread_lock_wait_time_seconds	Python thread lock waits

Kubernetes state (kube-state) — kube_* · pod_*

Metric	Description
kube_pod_info · kube_pod_status_phase · kube_pod_container_status_ready	Pod metadata, state, container readiness
kube_pod_container_resource_limits · kube_pod_container_resource_requests	Container requests and limits (CPU, memory)
kube_deployment_spec_replicas · kube_statefulset_replicas · kube_daemonset_status_desired_number_scheduled	Workload replicas
pod_count · pod_pending · pod_failed	Pod count, pending, failed
kube_node_info · kube_service_info	Node and service metadata

Note: the tables above list only the main metrics. Which detailed metrics exist — by language, protocol, GPU and so on — varies by environment, so always confirm what is actually available with label_values(__name__).

The supported syntax

The following is available in the default configuration. An administrator can turn some advanced functions off through a server option (see the note below).

Metric selectors and label matchers

Add label conditions in braces after the metric name to narrow the target.

Matcher	Meaning	Example
label="value"	Exact match	{namespace="prod"}
label!="value"	Not equal	{namespace!="kube-system"}
label=~"regex"	Regular expression match	{pod=~"web-.*"}
label!~"regex"	Regular expression non-match	{pod!~"canary-.*"}

Variables can be used alongside: {namespace="{{namespace}}"}. Multi-select variables are substituted automatically in the form namespace=~"a|b".

Aggregation operators

Gather several series into one. by(...) sets what to group by; without(...) sets which labels to drop.

Operator	Description
sum	Total
avg	Average
min / max	Minimum / maximum
count	The number of series
topk(k, ...) / bottomk(k, ...)	The top / bottom k
quantile(φ, ...)	Quantile
stddev / stdvar	Standard deviation / variance
group	Group existence (every value becomes 1)

For example sum by(namespace)(container_memory_rss) gives memory totals per namespace.

Binary operations and comparisons

• Arithmetic: + - * / %
• Comparison: > < >= <= == != — keeps only the series that satisfy the condition. Add the bool modifier to turn true/false into 0/1 values.
• Logical and set: and or unless

For example sum(rate_metric) / sum(total_metric) * 100 calculates a percentage.

Vector matching

Pairs different metrics by label for an operation.

Modifier	Description
on(labels)	Match on the listed labels only
ignoring(labels)	Match ignoring the listed labels
group_left(labels)	The left side is many-to-one (take labels from the right)
group_right(labels)	The right side is many-to-one

For example pod_metric * on(node) group_left(role) node_meta.

Functions

Mathematical functions

abs · ceil · floor · round · exp · ln · log2 · log10 · sqrt · sgn · clamp(v, min, max) · clamp_min(v, min) · clamp_max(v, max)

Label functions

• label_replace(v, dst, replacement, src, regex) — change a label's value, or create a new label, with a regular expression
• label_join(v, dst, sep, src1, src2, ...) — join several labels into a new one

Time and date functions

• time() — the time (in seconds) at each point
• timestamp(v) — the sample's timestamp
• hour(v) · minute(v) · day_of_week(v) · day_of_month(v) · days_in_month(v) · month(v) · year(v) — with an argument, computed from that value; without one, from each point in time (Korea Standard Time)

Range aggregation (_over_time)

Aggregates the values within a given time window. For example avg_over_time(metric[5m]).

avg_over_time · sum_over_time · min_over_time · max_over_time · count_over_time · last_over_time · present_over_time · stddev_over_time · stdvar_over_time · quantile_over_time

Note: the range window ([5m], say) must be at least as long as the query interval (step). A window shorter than the step produces inaccurate values and is treated as an error.

Other functions

• histogram_quantile(φ, ...) — a quantile (P95, say) from a histogram
• vector(s) — turns a scalar into a series with no labels
• scalar(v) — turns a single series into a scalar
• absent(v) · absent_over_time(m[5m]) — returns 1 where there is no value (detecting gaps)
• sort · sort_desc · sort_by_label — sorting is meaningless in a range query, so these pass through unchanged

Comparing against an earlier period (offset)

offset fetches the values of the same window in the past for comparison.

For example metric / (metric offset 1d) gives the ratio against a day ago.

Common patterns and their caveats

Do not wrap request rates and throughput in rate()

This is the most common trap. Metric aliases with a cumulative character, such as request counts, already have the rate applied. You must not wrap them again in rate() or increase().

• ✅ Correct: sum by(app_id)(container_http_requests_count{namespace="{{namespace}}"})
• ❌ Wrong: rate(container_http_requests_count[5m])

Apply only an aggregation such as sum by(...) on the outside.

Narrowing by namespace

Most metrics can be narrowed by namespace with the {namespace="{{namespace}}"} matcher. Using a variable re-queries the panel automatically when you change namespace in the header combo.

Response time percentiles

Get latency percentiles either by applying histogram_quantile to a histogram-based metric, or by approximating — dividing a load-weighted total (total request-seconds, say) by the request count. See the SLO and RED examples in the templates.

Tip: when building a new panel, check the query first with the Preview button in the panel editor dialog of custom dashboards. Errors and empty results are reported with a message.

What is not supported

The following are not currently supported.

Item	Alternative
rate() · increase()	Already applied to the metric alias — use it as it is, unwrapped
The @ modifier (a fixed point in time)	Ignored without an error (only offset is honoured)
count_values	Not supported
Subqueries [5m:1m]	Not supported
predict_linear · holt_winters	Use Forecast in the SRE report
Trigonometric functions (sin, cos and so on)	Not supported
Time masking such as metric and hour() >= 9	Not supported (time functions carry no labels, so masking is impossible)

Unsupported syntax is usually reported as an error at preview or output time; some of it (the @ modifier, for instance) is ignored silently.

Using variables

A variable is a filter value applied across several panels. Refer to it in a query as {{variable}}.

• query type variables look values up with label_values(metric, label). For example label_values(namespace).
• Dependent variables put the parent variable into a matcher to narrow the child list. For example label_values(container_info{namespace="{{namespace}}"}, pod).

For how to define and select variables, see the "Filtering with variables" section of custom dashboards.

Note: some special metrics (user estimation metrics stored under a composite key, for instance) may not be suitable as a source for label_values(). Where a variable list comes up empty, enumerate namespaces from an ordinary metric such as container_http_requests_count.

Frequently asked questions

The query returns nothing

• Check the metric name and label values for typos.
• Check that the matcher is not too narrow (that the {namespace="..."} value actually exists).
• Where you use a dependent variable, the parent variable has to be selected first.
• Use the Preview button to see the result and any error message.

The request rate looks strangely high or low

You have most likely wrapped a request count metric in rate(). The rate is already applied to that metric, so use sum by(...) alone, without rate().

I want to use predict_linear

Cache PromQL does not support forecasting functions. For forecasting future usage, use the Forecast tab of the SRE report, which offers linear regression, Holt-Winters, ARIMA and SARIMA.

An advanced function does not work

Mathematical functions, range aggregation and other advanced functions are on by default, but in environments where an administrator has disabled them through a server option you may see "this function is disabled". Ask your administrator.

Related documents

• Custom dashboards — building dashboards with PromQL panels and variables
• SRE report — the PromQL panels of custom reports, and forecasting
• Common chart guide — the shared components and controls of time-series charts