Skip to content

R7. PromQL Integration Reference

Diátaxis: Reference · Audience: operators / administrators (with PromQL or Grafana experience) ← Back to contents

If you are already comfortable with Grafana dashboards or PromQL, the WAS, JVM, system, WEB, and DBMS metrics OPENMARU APM collects can be queried directly with standard PromQL-compatible queries. Collection, storage, querying, and visualization all happen in APM alone, with no separate Prometheus server, exporter, or scrape configuration, and existing Grafana dashboard assets can be reused.

  • APM's fine-grained metrics as they are -- reach JVM, GC, thread, DB connection pool, and transaction metrics with standard PromQL
  • Rich labels -- filter and aggregate precisely with APM context such as instance_id, ip_addr, and agent_type
  • rate/increase are fast and accurate -- stored as a delta at collection time, so a query only sums
  • Relative time expressions -- intuitive forms such as now and now-1h are supported (a convenience standard Prometheus does not have)

For what each metric means, see R2. Chart Metric Reference and E3. What the Metrics Mean. To gather charts inside the console, H3. My Dashboard is enough -- PromQL is the tool for external integration and automation.

How to open it -- in a browser, http://{apm-server}/monitoring/api/v1/metric-explorer (the Metric Explorer UI). Sign in with an API access key -- go to the left menu ▸ Settings ▸ Users, press the edit user button, and create and copy the key.

Getting Started in 5 Minutes​

The simplest query -- one line gives the JVM heap usage (current value) of every WAS instance.

jvm_heap_heapUsed

To see only a particular instance, or only a particular IP range (regular expression):

jvm_heap_heapUsed{instance_id="apm-was-01"}
jvm_heap_heapUsed{ip_addr=~"192\\.168\\.80\\..*"}

Apply time functions to get the maximum over the last hour and the 5-minute average GC frequency:

max_over_time(jvm_heap_heapUsed[1h])
rate(jvm_gc_gcCount[5m])

Note Choose the exact metric name from the Metric Explorer dropdown, or check it with GET /api/v1/label/__name__/values (about 250 of them).

Querying a Specified Period​

A range query (/api/v1/query_range) takes the start and end times and a step (the data interval). Four time formats are supported.

FormatExampleNote
Relative timenow, now-1h, now-30m, now-1dThe most convenient -- units s/m/h/d
Unix epoch (seconds)1704067200shell date +%s
Unix epoch with a fraction1704067200.123Millisecond precision
RFC33392026-05-10T09:00:00+09:00Can include a time zone

Frequently used call examples:

# The last hour -- the most common case (step automatically 15s)
curl -G "http://{apm-server}/monitoring/api/v1/query_range" \
  --data-urlencode "query=jvm_heap_heapUsed" \
  --data-urlencode "start=now-1h" \
  --data-urlencode "end=now"

# The weekly trend over the last 7 days (step automatically 5m)
curl -G "http://{apm-server}/monitoring/api/v1/query_range" \
  --data-urlencode "query=max_over_time(cpu_usage_cpuUsage[1h])" \
  --data-urlencode "start=now-7d" \
  --data-urlencode "end=now"

# Specifying the step directly (15-second interval)
curl -G "http://{apm-server}/monitoring/api/v1/query_range" \
  --data-urlencode "query=rate(jvm_gc_gcCount[5m])" \
  --data-urlencode "start=now-30m" \
  --data-urlencode "end=now" \
  --data-urlencode "step=15s"

# A particular date range (KST)
curl -G "http://{apm-server}/monitoring/api/v1/query_range" \
  --data-urlencode "query=jvm_heap_heapUsed{agent_type=\"WAS\"}" \
  --data-urlencode "start=2026-05-10T09:00:00+09:00" \
  --data-urlencode "end=2026-05-11T18:00:00+09:00"

Without a step, it is calculated automatically from the query range:

Query rangeAutomatic stepData points (example)Suggested use
12 hours or less15s1 hour ≈ 240Real time · standard dashboards
12 hours to 2 days1m24 hours ≈ 1,440Day-to-day comparison
2 to 8 days5m7 days ≈ 2,016Weekly trend
8 to 21 days10mFortnightly trend
21 days or more30m30 days ≈ 1,440Monthly comparison

Note The raw data is stored every 2 seconds. A smaller step is denser but loads the server more -- start with a large step and reduce it when you need to.

If you need only one value at a single point in time, use an instant query (/api/v1/query):

# The heap usage now / one hour ago
curl -G "http://{apm-server}/monitoring/api/v1/query" \
  --data-urlencode "query=jvm_heap_heapUsed"
curl -G "http://{apm-server}/monitoring/api/v1/query" \
  --data-urlencode "query=jvm_heap_heapUsed" \
  --data-urlencode "time=now-1h"

Metric Names and Labels​

APM metrics are distinguished by five label dimensions. They are used directly in query filters and aggregation.

LabelMeaningExample values
__name__The metric name -- in the form {nameSpace}_{metricName}_{field}jvm_heap_heapUsed, cpu_usage_cpuUsage
ip_addrThe IP of the host the agent is installed on192.168.80.190
agent_typeThe agent typeWAS, SYS, DBMS, WEB
instance_idThe instance identifier (unique within the host)apm-was-01, nginx-1
namespaceThe metric groupjvm, transaction, cpu, memory

Prometheus-standard labels are generated automatically as well -- instance (= {ip_addr}:{instance_id}) and job (= agent_type).

For metric names, only one form has to be known: {nameSpace}_{metricName}_{field} (the trailing _field may be omitted -- omitting it selects the default field).

ExampleMeaning
jvm_heap_heapUsedThe used field of the JVM heap
jvm_heap_heapMaxThe maximum field of the same metric
cpu_usage_cpuUsageCPU utilization
cpu_usage_systemCpuUsageSystem-wide CPU utilization

All four standard label matchers are supported, and several conditions are combined with commas:

metric{label="value"}    # equal
metric{label!="value"}   # not equal
metric{label=~"regex"}   # regular expression match
metric{label!~"regex"}   # regular expression non-match

jvm_heap_heapUsed{ip_addr="192.168.80.190", instance_id=~"apm-was-.*", agent_type="WAS"}

How It Differs from Prometheus​

Almost all the syntax is identical; one thing differs -- how counters are stored.

AspectPrometheusOPENMARU APM
Stored counter valueA monotonically increasing cumulative value (total requests 1, 2, 3 …)The delta per unit time (17 in 2 seconds)
rate() calculation(last value − first value) ÷ periodsum of deltas ÷ period
increase() calculationlast value − first valuesum of deltas
Counter reset correctionRequiredNot required (a delta has no reset concept)

What the user sees is the same -- rate(jvm_gc_gcCount[5m]) is "the average GC count per second" on both. Only the internal calculation differs, and storing deltas up front makes it faster and more accurate.

The following are 100% compatible -- label matchers, aggregation operators (sum by and so on) with topk/bottomk/quantile, rate/irate/increase, the *_over_time function family, the maths functions, arithmetic and comparison operations, and the HTTP API specification.

Unsupported items and limitations:

ItemState
histogram_quantile()Not supported -- APM stores quantiles directly, so use fields such as avgRT, minRT, and maxRT straight away
Recording / alerting rulesNot supported
Subquery (<expr>[<range>:<step>])Limited support
predict_linear(), holt_winters()Not supported
Automatic rollup for long-range queriesNot applied -- for queries over several weeks, specifying a large step is recommended

Note Querying metric{...} without a function returns the last sample in the step window. If you need an exact sum or average over the window, write sum_over_time(...) or avg_over_time(...) explicitly.

Supported Functions​

All four standard data types (instant vector, range vector, scalar, string) are supported.

Rate / counter functions -- range vector → instant vector:

rate(metric[5m])       # average rate of change per second
irate(metric[5m])      # instantaneous rate from the last two samples
increase(metric[5m])   # the total increase over the window

The over_time family -- window statistics:

avg_over_time(metric[1h])    min_over_time(metric[1h])    max_over_time(metric[1h])
sum_over_time(metric[1h])    count_over_time(metric[1h])  last_over_time(metric[1h])

Aggregation operators -- sum, avg, min, max, count, stddev, stdvar, topk(N, …), bottomk(N, …), quantile(0.95, …), count_values, group. Group with a by or without clause:

avg by (instance) (jvm_heap_heapUsed)        # average per instance
sum without (user_key) (jvm_heap_heapUsed)   # group by everything except user_key
topk(5, rate(jvm_gc_gcCount[5m]))            # the top 5 instances by GC frequency

Maths, arithmetic, and comparison:

abs(metric)  ceil(metric)  floor(metric)  round(metric)

metric_a + metric_b
rate(metric[5m]) / 60
jvm_heap_heapUsed > 1024 * 1024 * 1024   # returns only the series over 1 GB

Metric Explorer​

A UI for running PromQL interactively in a browser -- reach it at the address in How to open it above.

AreaFunction
Metric selection dropdownAutomatic discovery of registered metrics (by __name__)
PromQL query inputType it directly -- choosing from the dropdown fills it in automatically
Label filter panelDropdowns per ip_addr, agent_type, instance_id, and namespace
Aggregate function selectionsum, avg, topk, quantile, and others
Time rangeQuick buttons from the last 5m to 7d, plus a custom range
StepAuto / 15s / 1m / 5m / 10m / 30m / 1h
Result visualizationLine, area, and bar charts plus a table of raw values
Copy API URLCopies the current query as an /api/v1/query_range URL -- for external integration such as Grafana

Making your first chart in 30 seconds:

  1. Select jvm_heap_heapUsed in the metric dropdown
  2. Select agent_type=WAS in the label filter
  3. Click the 1h time range and leave Step at Auto
  4. Click Run → a graph of one hour of heap usage across the WAS instances appears

Note Auto is usually enough for Step. A small step with a long range loads the server heavily.

Grafana integration -- the same API can be connected as a Prometheus data source in Grafana:

URL: http://{apm-server}/monitoring
Access: Server (default)

Grafana calls /api/v1/query, /api/v1/query_range, /api/v1/labels, and the rest automatically -- reuse existing dashboard and alerting assets as they are.

HTTP API Summary​

MethodPathPurpose
GET / POST/api/v1/queryInstant (single point in time) query
GET / POST/api/v1/query_rangeRange (time range) query
GET/api/v1/seriesSeries metadata
GET/api/v1/labelsThe list of label names
GET/api/v1/label/{name}/valuesThe list of values for a particular label
GET/api/v1/metadataMetric metadata
GET/api/v1/metric-explorerThe Metric Explorer UI

Parameters -- query (required), time for instant, and start, end, and step for range (calculated automatically when omitted). Exploratory calls:

# The full list of available metrics (~250)
curl "http://{apm-server}/monitoring/api/v1/label/__name__/values"

# The list of label names / the values of a particular label
curl "http://{apm-server}/monitoring/api/v1/labels"
curl "http://{apm-server}/monitoring/api/v1/label/agent_type/values"

A Collection of Frequently Used Queries​

The metric names are examples following the general pattern -- check the exact names for your environment in the Metric Explorer.

JVM / WAS:

jvm_heap_heapUsed{agent_type="WAS"}              # current heap usage across all WAS
avg_over_time(jvm_heap_heapUsed[5m])             # 5-minute average heap usage
topk(5, jvm_heap_heapUsed)                       # the top 5 instances by heap usage
(jvm_heap_heapUsed / jvm_heap_heapMax) > 0.8     # heap utilization over 80%
sum_over_time(jvm_gc_gcTime[1h])                 # total GC time (1 hour)
rate(jvm_gc_gcCount[5m])                         # GC count per second
datasource_pool_active{agent_type="WAS"}         # DataSource connections in use

System:

topk(10, cpu_usage_cpuUsage{agent_type="SYS"})   # the top 10 hosts by CPU
cpu_usage_systemCpuUsage                         # system-wide CPU
avg_over_time(cpu_usage_cpuUsage[1h])            # 1-hour average CPU per instance

Aggregating per host:

avg by (ip_addr) (jvm_heap_heapUsed)             # average heap per host
count by (agent_type) (jvm_heap_heapUsed)        # instance count per agent_type
{ip_addr="192.168.80.190"}                       # every series for a particular IP

When It Does Not Work​

SymptomWhat to check
You do not know the metric nameCheck the actual name in the Metric Explorer dropdown or at /api/v1/label/__name__/values
The result is emptyA typo in a label value -- check the possible values at /api/v1/label/{name}/values, and check the case of agent_type
Metric Explorer sign-in failsAn API access key is required -- create and copy the key under Settings ▸ Users
A rate() value is not what you expectAPM stores deltas, so there is no reset correction -- check you have not confused it with a query without a function
The sum or average does not matchA query without a function gives the last sample -- write sum_over_time or avg_over_time explicitly
histogram_quantile does not workNot supported -- query quantile fields such as avgRT and maxRT directly
A long-range query is slowSpecify a large step (30m to 1h) -- a small step over a long range is heavy

This chapter also appears with the same content as chapter 7 of the OPENMARU APM API Integration Guide. To read it alongside the JSON APIs that use a session cookie (graph data, configuration, user management), see that guide.