Skip to content

4.2. Applications

The full list of monitored applications, and detailed observability data for each one.

The applications screen

Overview

The Applications menu shows every application OPENMARU Observability monitors, on one screen. An application is the basic unit of monitoring; in a Kubernetes environment that means a workload such as a Deployment, StatefulSet or DaemonSet.

Read each application's current state and headline metrics quickly, then click one to see detailed reports — SLOs (service level objectives), CPU, memory, logs, distributed traces, profiling and more.

Click Applications in the left sidebar to reach this screen.


The application list

Screen layout

The top of the list screen carries the filters, the view toggle and the status legend, with the application list below.

AreaDescription
Top headerPage title, application filters (namespace, category, search)
View toggleSwitches between list view and card view
Status legendShows the application count per state, and filters by it
List areaList view: a table. Card view: a grid of cards
PaginationItems per page (10, 20, 50, 100, All) and page navigation

Status legend

The status legend filter

The status legend in the toolbar at the top summarises every application by state.

StateMeaning
SLO breachThe application has passed an SLO threshold. It needs immediate action.
WarningThe application is in a warning state and needs attention
Errors in logsAn error pattern was detected in the application's logs
Security threatAn attack pattern was detected against the application
Integration neededNo agent connection, so no data is being collected for the application
OKThe application is running normally

Click an entry in the status legend to filter down to applications in that state. Several states can be selected at once; the selected entries are highlighted and the rest dim. Click again to clear the filter.

Tip: check the count of applications in an SLO breach or warning state first, then click that state to concentrate on the ones with problems.


Main features

Filters

Use the filters at the top right to narrow the list by namespace, category and application name. Combine conditions to show only the applications you want.

Hiding applications

Temporary applications that appeared briefly through a misconfiguration (unknown_service, say), or applications you no longer want to see, can be hidden globally (a soft hide) to clear noise from the list and the topology map. A hidden application is not deleted; it is gathered on the Hidden applications screen and can be shown again at any time.

List view

List view

Click the list icon in the view toggle at the left of the toolbar to switch to list view, which suits comparing many applications at once.

Click a column header to sort by it, and click several columns in turn to sort by more than one.

The main columns in list view are as follows.

ColumnDescription
ApplicationThe name and namespace. Click it to open the detail page.
KindThe Kubernetes workload kind (Deployment, StatefulSet, DaemonSet and so on)
TypeThe application's technology stack (Java, Python, Redis, Kafka and so on)
ErrorsThe error rate against the SLO
Response timeThe response time against the SLO
UpstreamHow many upstream services this application calls
InstancesHow many instances are running
RestartsHow many times instances have restarted
CPUCPU usage
MemoryMemory usage
I/O loadDisk I/O load
Disk spaceDisk usage
NetworkNetwork state
DNSDNS query state
Log patternsThe recent count of log error patterns, with a trend chart

Note: where a figure such as errors or response time is highlighted in red or orange, that item has passed an inspection threshold. Click the value to jump straight to the relevant detail tab.

Card view

Card view

Click the grid icon in the view toggle to switch to card view, which shows each application's state and headline metrics as a card. Each card carries the application name, a status icon and sparkline charts of the main metrics; click a card to open the detail page.

Your choice of view is remembered for your next visit.

Card view lets you choose the sort order.

Sort byDescription
StatusCritical → warning → log errors → integration needed → healthy
ApplicationAlphabetically by name
ErrorsBy error count, descending
Response timeBy response time, descending
RestartsBy restart count, descending
CPUBy CPU usage, descending
MemoryBy memory usage, descending
LogsBy log error count, descending

Below the cards are the page navigation buttons and the items-per-page setting (10 / 20 / 50 / 100 / all).

Tip: sorting by status puts the applications with problems at the top, where you can take them in at a glance.


Application details

The application detail screen

Click an application's name in the list to open its detail page.

Screen layout

The detail page is made up of the following areas.

AreaDescription
Top headerApplication name, namespace, kind, and a link to the topology map
Time rangeThe time range currently being queried (three days at most)
Dependency map (AppMap)A visualisation of the services this application connects to
Report tabsPer-tab detail: SLO, instances, CPU, memory, logs and more
Inspection statusWhether each report's inspection conditions are met
Chart dashboardThe metric chart widgets for the selected tab

The top header carries the following information and features.

  • Application name: the name of the monitored target
  • ns: the namespace it belongs to
  • kind: the Kubernetes workload kind
  • Open topology map button: opens the topology map with this application highlighted.
  • Open COP Console button: an external link that takes you straight to the Kubernetes resources in COP Console. (Shown only where COP Console is configured.)
  • Back: returns to the previous application list screen.

The query time range

The top of the detail screen shows the time range currently being queried. Where the range is longer than three days, a 3-day limit badge appears and only the last three days of data are shown. Hover over the badge to see the range actually being queried.

Where there is no data in the selected range, a dialog appears; Go to the last hour shows you the most recent data, and Go to the application list returns you to the list.

Note: changing the global time range in the time picker in the top bar updates the detail page's data too.

Dependency map (AppMap)

The dependency map

The Show map / Hide map button at the top of the detail screen toggles the dependency map, which visualises how this application relates to the services it connects to.

  • Upstream: the services this application calls
  • Downstream: the services that call this application

Click a service node on the dependency map to see its details or view it on the topology map.


Report tabs

The detail page offers several report tabs. Which tabs appear depends on the kinds of data the agent collected. A status light (LED) before each tab name tells you that report's current state at a glance.

  • Green: healthy
  • Orange: an anomaly at warning level
  • Red: a serious problem
  • Grey: no data, or not configured

SLO tab

The SLO tab

Where SLO data exists, the SLO tab is selected by default when you open the detail page (where the SLO state is unknown — an app with no requests, say — another tab may be chosen instead). It charts current performance against the availability and response time targets.

MetricDescription
Availability SLOThe proportion of requests handled without error, and the target
Latency SLOThe proportion of requests handled within the target response time
Error budgetHow many errors are permitted while still meeting the SLO target
Error rate (%) chartThe error rate (%) trend is drawn below the errors-per-second chart (periods with no errors appear as a 0% line)

Check the current state against the configured thresholds and see whether the SLO has been breached.

Instances tab

Lists the individual instances (pods) that make up this application, along with instance count and restart trends.

The instances tab — availability and restart inspections, the instance list, and instance count and restart charts
ItemDescription
InspectionsWhether the instance availability (too few available instances) and restart (container restarts occurring) conditions are met
Instance listEach instance's (pod's) state (running, pending and so on), restart count, IP address and the node it landed on
Instance count chartThe trend in actual (up) and desired instances
Restart count chartThe cumulative container restart count per instance

The instance count chart carries deployment marks (vertical dotted lines). A mark is placed wherever a new version is deployed or a rollout occurs; hover over one and a tooltip gives the deployment type, the application and version (ReplicaSet, image tag) and the deployment time. Putting deployment times alongside changes in instance count and performance tells you immediately whether a particular deployment caused a problem.

Instance count over time — a deployment mark (hover tooltip: deployment type, app, version, time)

The instance count chart also marks scale-up (↑) and scale-down (↓) events driven by HPA and CronHPA, together with the reason (a CPU or memory target exceeded, or a schedule). This separates changes caused by autoscaling from manual scaling and redeployments, so you can follow the autoscaler's behaviour.

Instance count over time — HPA scale-up (↑) and scale-down (↓) marks

CPU tab

Several charts covering the application's CPU usage.

The CPU tab — node and container CPU inspections, and CPU usage, load average, CPU delay and throttling charts
MetricDescription
CPU usage (cores)CPU usage in cores over time, per container and per instance
CPU delayTime spent ready to run but waiting to be given CPU — an indicator of contention on the node
Throttled timeTime execution was held back by the CPU limit. A large value suggests raising the CPU limit
Node CPU usageThe CPU usage of the whole node this application runs on

Inspections: warns when node CPU is overloaded, or container CPU usage passes its threshold.

Memory tab

Several charts covering the application's memory usage.

The memory tab — OOM and leak inspections, and memory usage (RSS), node memory and top-consuming process charts
MetricDescription
Memory usage (RSS)Actual physical memory usage (RSS) over time, per container
Node memory usageThe memory usage of the whole node it runs on (including unreclaimable areas)
Top memory processesWhich processes are consuming the node's memory (a stacked area chart)
OOM killsHow many times a container was killed for running out of memory

Inspections: warns when containers are killed for lack of memory (OOM), or the hourly growth in memory usage (suggesting a leak) passes its threshold.

Logs tab

The logs tab

Query the logs this application produced.

What it offers:

  • Source selection: choose the log collection source (agent or otel) to see each source's logs.
  • Keyword search: type a keyword into the search field and press Query to find particular logs.
  • Severity filter: use the Error, Warning, Info and other level checkboxes to show only the severities you want.
  • Log pattern view: shows log patterns, produced by grouping structurally similar log messages automatically. Useful for taking in repeated error patterns at a glance.
  • Full log view: switch from the pattern view to the full view to see individual log messages in time order.
  • Sorting: switch between newest first and oldest first.
Analysing logs with CogentAI

The logs tab lets you use CogentAI analysis to interpret log messages automatically and get a diagnosis of the cause of an error along with suggested fixes.

Requesting analysis from the log list:

  1. Click the CogentAI icon beside the log row you want analysed.
  2. The CogentAI widget opens at the bottom right of the screen and that log message is passed to the AI as an analysis request.
  3. The AI analyses what the log means, why it occurred and what action it recommends, and shows the result.

Requesting analysis from the log detail dialog:

  1. Click a log row in the list to open the detail dialog.
  2. Click the CogentAI Insight button at the top of the dialog.
  3. A more detailed analysis of that log appears in the CogentAI widget.

Requesting analysis from a log pattern:

  1. In the log pattern view, click the CogentAI icon on a pattern card, or the CogentAI Insight button in the pattern detail dialog.
  2. The AI's analysis of that log pattern appears in the widget.

Tip: running CogentAI over WARN and ERROR level logs gets you concrete guidance, including the steps to resolve the problem and the commands involved.

Analysing charts with CogentAI

Each metric chart also carries a CogentAI Insight icon at its top right. Click it and the AI analyses that chart's data, explaining anomalous patterns, interpreting trends and recommending action.

  1. Click the CogentAI Insight icon at the top right of the metric chart you want analysed.
  2. The CogentAI widget opens and that chart's metric data is passed as an analysis request.
  3. The AI interprets the chart data and offers its insight.
  4. Type a follow-up question into the widget to ask for more.

Note: CogentAI features are available only where an administrator has enabled the CogentAI widget.

Deployments tab

A table of this application's deployment history. The change in metrics either side of each deployment is summarised automatically, so you can see at a glance which deployment affected performance or stability.

The deployments tab — the deployment history table (name, time, elapsed, change summary)

The table has the following columns.

ColumnDescription
DeploymentThe version deployed (ReplicaSet hash, image tag) and how long that version lasted (until the next deployment)
Deployment timeWhen the deployment was detected
DeployedHow long ago it was (for example "3 hours ago")
SummaryAn automatic summary of how availability, CPU, memory and log errors changed against the previous deployment. Improvements in green, regressions in red

Inspections: warns when a rollout takes longer than its threshold (a deployment that does not finish).

Tip: click the icon beside an item in the summary to jump to that metric's before-and-after chart and see the change in detail. "CPU usage: 8.3% different from the previous deployment", for instance, means this deployment increased CPU usage.

Profiling tab

The profiling tab

The profiling tab visualises the application's CPU and memory usage patterns as a flame graph — a chart that visualises the function call stack and the time spent in each frame.

What it offers:

  • Status: the profiling collection state is shown at the top as OK, WARN or CRIT.
  • Profile type: click a profile type button, such as CPU or memory, to switch.
  • Time-series chart: shows how profile data trends over time. Drag to select a period on the chart and the flame graph for it appears, which you can compare against the preceding period.
  • Flame graph: the wider a block, the more resource that function is using. Click a block to zoom in and inspect the call stack below it.
  • Instance selection: select one instance's profile to compare.

Tip: dragging a period on the profile chart not only gives you the flame graph for it but also lets you compare it against the preceding period of the same length.

Profiling Go applications (pprof)

Profiling data is collected along two paths.

  • eBPF continuous profiling (automatic): the node agent collects CPU profiles continuously, with no code changes (CPU (eBPF) in the type picker).
  • pprof scraping (Go, via annotations): the cluster agent collects the /debug/pprof endpoint a Go application exposes. As well as CPU, this gives you heap memory, goroutine, mutex and block profiles.

To connect pprof profiling to a Go application, expose net/http/pprof in the app and add the following annotations to the pod.

Go pprof profiling — the type picker (alloc, in-use, block contention/delay, mutex contention/delay, CPU, goroutines) and the flame graph
# Pod template (spec.template.metadata.annotations)
openmaru.io/profile-scrape: "true" # enables profile collection (alias pyroscope.io/scrape)
openmaru.io/profile-port: "6060" # the port pprof listens on (alias pyroscope.io/port)

The cluster agent collects these five profiles from /debug/pprof/.

Typepprof pathContents
CPUprofileThe CPU usage profile
MemoryheapHeap allocations and in-use objects and sizes
GoroutinesgoroutineGoroutine stacks
MutexmutexLock contention
BlockblockBlocking points

Note: the mutex and block profiles are off by default in Go, so the app has to enable them with runtime.SetMutexProfileFraction(n>0) and runtime.SetBlockProfileRate(n>0). Applications that do not expose pprof (the stock otel-demo Go services, for example) are not collected this way. For a sample that emits all five, see test/go-pprof-demo/.


Distributed tracing tab

The distributed tracing tab

The distributed tracing tab analyses this application's trace data through a heatmap and a trace list. A trace is a record of the path one request took through the several services of a distributed system. The tab supports the whole workflow: select a region of interest on the heatmap, look at individual traces in the list, then drill down into the detailed call flow with the span waterfall chart.

Step 1: check the trace state and choose a source

When you open the distributed tracing tab, the collection state of the trace data is shown at the top.

  • Status: a badge reading OK (collecting normally), WARN or CRIT, with a message describing the situation.
  • Source switching: where the agent supports several sources, a source switch appears.
    • agent: eBPF-based automatic trace data, collected without code changes.
    • otel: OpenTelemetry SDK trace data, which carries finer-grained span information.
  • Connection settings: click the settings (gear) icon in the status area to open the dialog for connecting this application to an OpenTelemetry service.
Step 2: narrow the traces with filters

Three filter toggle buttons sit above the heatmap.

FilterDescription
Show error tracesShows only traces where an error occurred
Show latency SLO breachesShows only traces that passed the response time SLO threshold
Show all tracesShows all trace data

Click a filter button to turn it on; the heatmap and the trace list below refresh to match.

Tip: when analysing a spike in errors, turn on Show error traces first to concentrate on the traces involved.

Step 3: select a region of interest on the heatmap

The heatmap visualises the distribution of time (X axis) against response time (Y axis) as colour density. The darker the colour, the more requests in that response time range at that moment.

Selecting a region on the heatmap:

  1. Click and drag across the region you want to investigate.
  2. The selected region is highlighted and the trace list below refreshes to the data within it.
  3. To change the selection, drag another region or click a filter toggle again.

Reading the heatmap:

  • Points concentrated towards the top mean many requests with long response times.
  • Red points at a particular moment mean errors occurred then.
  • In a healthy state, most points sit towards the bottom (low response times).

Tip: dragging over a region where response times jumped (points concentrated at the top of the heatmap) gets you straight to the traces involved.

Step 4: look at individual traces in the list

The trace list appears as a table below the heatmap. Where you selected a region on the heatmap, only the traces within it are listed.

ColumnDescription
Trace IDThe trace's unique identifier (first 8 characters shown). Click it to open the span detail dialog.
Start timeWhen the trace began
DirectionInbound (a request coming into this application from outside) or outbound (a request this application sent out)
ClientThe name of the service that sent the request
StatusOK or error, with the status message
Response timeHow long the whole trace took (milliseconds)
HTTP methodThe request's HTTP method or the operation name. Click it to open the span detail dialog.
DetailsThe request URL, query and so on. A copy button copies it to the clipboard.

Click a column header to sort, including sorting by more than one column. The items per page can be set below the table.

Note: where the result is large, a message about the maximum row limit may appear. Select a narrower region on the heatmap, or add a filter, to reduce the scope.

Step 5: analyse the detailed flow with the span waterfall chart

Click a trace ID or an HTTP method in the trace list and that trace's span waterfall chart opens in a dialog. A span represents a single unit of work within a distributed trace.

The top of the dialog:

The dialog header carries the following.

  • Trace ID: the full trace ID, with a copy button
  • Client: the name of the service that sent the request
  • Status: an OK or error badge
  • Response time: how long the whole trace took (milliseconds)

Reading the span waterfall chart:

The span waterfall chart shows visually the order in which a request passed through each service and how long each step took.

  • A horizontal bar starts where the span started, and its length is how long the span took.
  • Each span's service name gets its own colour.
  • Child spans are indented to express the hierarchy. Click a parent span's name to collapse or expand its children.
  • Spans where an error occurred carry an error icon.
  • Switch between Show subtrace and Show full trace to see just one service's child spans or the whole trace.

Span details:

Click a span to see its details.

ItemDescription
NameThe span's operation name
ServiceThe service the span belongs to
Start timeWhen the span started
Response timeHow long the span took (milliseconds)
StatusOK or error, with the message
Span IDThe span's unique identifier (copyable)
AttributesThe attributes attached to the span (HTTP URL, database query and so on)
EventsEvents that occurred while the span ran (an exception stack, for example)

The span type is shown as a badge (HTTP, gRPC, Kafka, Redis, MongoDB, PostgreSQL, MySQL and so on).

Worked examples with distributed tracing

Here are the common ways to analyse a problem with the distributed tracing tab.

Finding slow requests:

  1. Turn off Show error traces and turn on Show all traces.
  2. Drag to select the region of high response times (the upper part) on the heatmap.
  3. Click the Response time column in the trace list to sort descending.
  4. Click the trace ID of the slowest trace to see its span waterfall chart.
  5. Click the span with the longest bar to see which service and which operation is the bottleneck.

Analysing error traces:

  1. Turn on Show error traces.
  2. Drag to select the period where the errors are concentrated on the heatmap.
  3. Find a trace with an error status in the list and click its trace ID.
  4. Find the span carrying an error icon on the span waterfall chart and click it.
  5. Read the exception stack under Events in the span details to establish the cause.
The distributed tracing workflow in summary

Here is the whole flow for analysing a problem from the distributed tracing tab of the application details.

  1. Check the status: read the trace collection state from the status badge at the top.
  2. Choose a source: select agent (eBPF) or otel (OpenTelemetry) as the source to analyse.
  3. Apply filters: turn on the filters you need — error traces, SLO breaches — to narrow the scope.
  4. Select a heatmap region: drag to select the region with high response times or errors.
  5. Look at the trace list: identify the slow traces and error traces within the selected region.
  6. Analyse the span waterfall chart: click a trace ID and see which service and which span took the time, or where the error occurred.
  7. Check the span details: click the problem span and read its detailed attributes — HTTP URL, database query, error message.

Tip: to analyse traces across everything, use the Distributed tracing menu in the sidebar. The distributed tracing tab in the application details shows only traces involving that application.


Domain names tab

The DNS lookup metrics this application produced.

The domain names tab — DNS response time and error inspections, the per-domain query table, and query type and error charts
MetricDescription
DNS query response timeThe average response time of domain lookups
DNS error rateThe proportion of lookups that failed (NXDOMAIN, SERVFAIL and so on)
Distribution by query type and domainThe record types looked up (A, AAAA, CNAME and so on) and the request volume per target domain

DNS latency and errors are an early signal of latency in external service integrations, or of a service discovery failure.

Network tab

L4 (TCP) network metrics for this application's upstream and downstream connections.

The network tab — RTT and TCP connection inspections, and RTT, connection wait, active connection and connection attempt charts (per upstream)
MetricDescription
Network RTT (round-trip time)The round-trip latency to the peer service
TCP connection latencyHow long establishing a new TCP connection took
Active TCP connectionsHow many connections are currently held
TCP connection attempts and failures (per second)Connection attempts and failed connections per second — a spike in failures signals a fault in the target service
Traffic (bytes per second)Inbound and outbound bandwidth
TCP retransmissions (segments per second)Retransmissions caused by packet loss — an indicator of degraded network quality

Security tab

Shows the HTTP intrusion and attack attempts detected against this application. The node agent detects attack patterns in eBPF L7 traffic — SQL injection, XSS, path traversal, remote code execution (RCE), scanners — and shows them with the attack type, severity and source IP address.

The security tab — traffic spike, brute force, error rate and security pattern inspections, and security event and request rate charts
ItemDescription
InspectionsTraffic spike (request rate N times normal), brute force (the proportion of 401/403 responses), error rate spike (5xx), security pattern detection (attack event count)
Security event chartDetections per second by type (SQLi, XSS, RCE, path traversal and so on) — the example above is a healthy state with no attacks; where attacks are detected the chart fills with a colour per type
Request rate chartErrors as a proportion of all requests

This tab shows security events from one application's point of view. For live threat status across the whole cluster, the distribution of attack sources by country, and security threat reports, see the Security attack detection screen.

Database and middleware tabs

Where a database or cache type is detected, a dedicated report tab is added. The basic metrics collected from eBPF L7 traffic (query response time, throughput, error rate) appear from installation alone; complete the additional database monitoring integration described below and the internal database metrics appear as well.

PostgreSQL tab

The PostgreSQL tab — inspections (availability, response time, replication lag, connections), instances, and query response time and queries-per-second charts
MetricDescription
Average query latencyThe average query response time
Queries per secondThroughput
ConnectionsConnections by state
Idle transactionsTransactions left open and idle
Locked queriesQueries waiting on a lock
Blocking queriesQueries making others wait (sorted by the number waiting)
Queries by total timeQueries with the largest cumulative execution time
Queries by I/O timeQueries with the largest disk I/O time
Replication lagThe lag behind the replica (bytes)

Inspections: warns when availability, query latency, replication lag or connection count passes its threshold.

MySQL tab (shown on screen as Mysql)

The MySQL tab — availability, replication state, replication lag and connection inspections, instances (version), and query count and average response time charts
MetricDescription
Queries per secondThroughput
Average latencyThe average query response time
Queries by total timeQueries with the largest cumulative execution time
I/O time by tableDisk I/O time per table
TrafficBytes sent and received per second (read/write)
Slow queriesSlow queries per second
ConnectionsConnections by state
New connectionsNew connections created per second
Replication lagThe lag behind the replica (seconds)

The same applies to MySQL-compatible databases such as MariaDB.

Inspections: warns when availability, replication state, replication lag or connection count passes its threshold.

Redis tab

The Redis tab — inspections (availability, response time), instances (role, version), and average response time and queries-per-second-by-command charts
MetricDescription
Average latencyThe average command response time
Queries by commandThroughput per second by command type (GET, SET and so on)

The same applies to Redis-compatible caches such as KeyDB, Valkey and Dragonfly.

Inspections: warns when availability or latency passes its threshold.

MongoDB tab

The MongoDB tab — availability and replication lag inspections, instances (version), and queries-per-second and average response time charts
MetricDescription
Queries per secondThroughput
Average latencyThe average query response time
Replication lagThe lag behind the replica set (seconds)

Inspections: warns when availability or replication lag passes its threshold.

Memcached tab

The Memcached tab — availability inspection, instances (version, capacity), and commands-per-second-by-command, hit rate and eviction charts
MetricDescription
CommandsThroughput per second by command type (get, set and so on)
Hit rateThe proportion of cache lookups that succeeded (%)
Items evictedItems evicted per second for lack of memory

Inspections: warns when availability is impaired.

Note: dedicated database tabs are created only for the PostgreSQL, MySQL, Redis, MongoDB and Memcached types. Other data stores such as Cassandra, Elasticsearch and Kafka get no dedicated tab; their eBPF L7 traffic metrics (response time, throughput, errors) can be seen in the general request charts on the SLO tab and elsewhere.

Runtime tabs

Where an application's technology stack (its runtime) is detected, a dedicated runtime report tab is added showing the runtime's internal metrics. The tab is enabled either by the agent's automatic runtime detection or by OpenTelemetry instrumentation.

JVM tab — Java applications

The JVM tab — instances (Java version), heap memory, GC time, JVM pause time and metaspace
MetricDescription
Heap sizeHeap memory used and its ceiling (total). Used approaching total risks OutOfMemory
GC timeTime spent in garbage collection (seconds per second), broken down by GC type
Safepoint timeTime spent at safepoints, where every thread stops. Long safepoints mean stop-the-world pauses
MetaspaceThe size of the class metadata area
Live threadsLive, daemon and peak thread counts
Threads startedNew threads created per second
Lock contentionThread lock contention events per second
Loaded classesHow many classes are loaded

Inspections: warns when JVM availability or safepoint time passes its threshold.

.NET tab

The .NET tab — instances (runtime version), heap memory by generation, GC, allocation rate and exception charts
MetricDescription
Heap sizeHeap size per generation (cumulative)
GC collectionsGarbage collections per second
AllocationThe amount of memory allocated
ExceptionsExceptions raised per second
Heap fragmentationThe heap fragmentation ratio (%)
Thread pool queue sizeThe size of the thread pool's waiting queue
Thread pool sizeHow many threads are in the pool
Thread pool completedWork items completed per second
Monitor lock contentionsMonitor lock contention events

Inspections: checks .NET availability.

Python tab

The Python tab — the GIL (global interpreter lock) wait time chart
MetricDescription
GIL waiting timeTime spent waiting to acquire the GIL (global interpreter lock), in seconds per second. A high value means CPU-bound multithreaded contention is degrading parallel throughput

Inspections: warns when GIL wait time passes its threshold.

Note: a runtime tab appears only on applications where that runtime was detected (by the agent, or through OpenTelemetry instrumentation). For how to integrate OpenTelemetry, see distributed tracing.

Storage tab

Shown conditionally on applications that use persistent volumes (PVs, PVCs and so on). It carries storage metrics such as volume usage, I/O throughput and I/O latency.

The storage tab — disk I/O load and disk usage inspections, and the per-volume disk space (used/total) chart

The following metrics are provided per volume (mount).

MetricDescription
Disk spaceUsed and total capacity per volume. Warns when the usage ratio (%) passes its threshold
IOPSRead and write operations per second, per volume
BandwidthRead and write throughput per second (bytes/s), per volume
I/O latencyThe average wait for I/O to complete (await), per volume
I/O loadThe volume's total I/O latency (seconds per second). The higher the value, the closer the disk is to saturation
I/O utilizationThe proportion of time the disk is busy with I/O (%)

The volume list table summarises latency, I/O load, space and device per volume.

Inspections: warns when disk space usage (above 80% by default) or disk I/O load passes its threshold.

How NFS volumes are shown (important): for NFS volumes (and other network filesystems such as CephFS), only disk space (capacity and usage) is shown; per-block-device I/O throughput metrics are not, because a network volume has no local block device. Furthermore, the capacity and usage shown are for the whole NFS server export (the shared storage), not for the individual PVC. A PVC requested at 1Gi will still show the export's whole capacity (536 GB, say) and the total usage stored in it. This is because subdirectory-style NFS provisioners (nfs-client and others) do not apply a per-PVC quota, so the operating system can only report the shared export's statistics for the mount; other tools such as kubelet behave the same way. For block-device-backed PVCs (local volumes and most CSI storage), the mount is the PVC, so capacity and usage are exact.

Note: the storage tab appears only on applications that use persistent volumes. Applications that do not use them have no such tab.

GPU tab

Shown conditionally on applications that use a GPU. It carries GPU metrics such as GPU utilization and GPU memory usage.

The GPU tab — GPU utilization, GPU memory usage and other GPU metric charts

The table at the top lists the GPUs the application uses (GPU, name, vRAM, node, instance), and the following metrics are provided.

MetricDescription
GPU usageThe GPU compute utilization the application holds (%)
GPU memory usageThe proportion of GPU memory the application holds (%)
GPU usage — average/peakThe average and peak GPU utilization across several GPUs and instances
GPU memory usage — average/peakThe average and peak GPU memory utilization
GPU consumersEach container's share of the same GPU's utilization (stacked)
GPU memory consumersEach container's share of the same GPU's memory (stacked)

Note: the GPU tab appears only on applications that use a GPU. Applications that do not use one have no such tab.

Inspection status

The top of each report tab shows whether that area's inspection conditions are met. Where an item has passed a threshold, it is marked warning or critical, and you can look at the inspection condition in detail or adjust its settings.

  • Green left border: the inspection criteria are met (OK)
  • Yellow left border: a threshold passed at warning level
  • Red left border: a threshold passed at critical level

Additional database monitoring integration (instrumentation)

For databases such as PostgreSQL, MySQL, Redis, MongoDB and Memcached, eBPF-based L7 traffic (query response time, error rate) is collected automatically from installation alone. Collecting the internal database metrics as well — connection counts, active queries, cache hit rates, replication lag, slow queries — requires an instrumentation step that tells the system how to connect to the database.

Where an application is detected as a database type, an instrumentation notice appears in the inspection status area of the detail screen. Click Connection Configure and a dialog opens offering the two approaches below.

Database connection settings — prerequisites, and the Kubernetes annotation and manual configuration tabs

Prerequisite — a dedicated monitoring account

Prepare an account on the database with monitoring privileges first.

  • PostgreSQL: requires the pg_monitor role and the pg_stat_statements extension (the extension must be loaded through the shared_preload_libraries server setting).

    create role openmaru with login password '<PASSWORD>';
    grant pg_monitor to openmaru;
    create extension pg_stat_statements;
  • MySQL: requires an account with the following privileges.

    GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'openmaru'@'%';
  • Redis / MongoDB / Memcached: prepare an account (or password) with read access.

openmaru-observ-cluster-agent discovers pods carrying the openmaru.io/<type>-scrape: "true" annotation automatically and collects their metrics. The annotation has to go on the pod template (spec.template.metadata.annotations), not on a parent object such as a Deployment or StatefulSet.

A PostgreSQL example:

openmaru.io/postgres-scrape: "true"
openmaru.io/postgres-scrape-port: "5432"
# Credentials — (A) inline
openmaru.io/postgres-scrape-credentials-username: "openmaru"
openmaru.io/postgres-scrape-credentials-password: "<PASSWORD>"
# Credentials — (B) referencing a Kubernetes Secret (an alternative to A)
openmaru.io/postgres-scrape-credentials-secret-name: "postgres-secret"
openmaru.io/postgres-scrape-credentials-secret-username-key: "username"
openmaru.io/postgres-scrape-credentials-secret-password-key: "password"
openmaru.io/postgres-scrape-param-sslmode: "disable"

Other databases differ only in the prefix and the default port; the remaining keys (-credentials-*, -secret-*) follow the same rules.

TypeScrape annotationDefault portExtra parameters
PostgreSQLopenmaru.io/postgres-scrape5432-param-sslmode (for example disable)
MySQLopenmaru.io/mysql-scrape3306-param-tls (true/false/skip-verify/preferred)
Redisopenmaru.io/redis-scrape6379The username is optional (set -credentials-username where ACLs are in use); password
MongoDBopenmaru.io/mongodb-scrape27017
Memcachedopenmaru.io/memcached-scrape11211

Approach 2 — manual configuration (UI)

Where Kubernetes annotations are unavailable, or the database sits outside the cluster, enter the port, username and password directly on the Manual Configuration tab of the connection settings dialog (including sslmode for PostgreSQL and the tls option for MySQL) and save.

Note: it can take several minutes after configuration for the database-specific metrics to appear. The cluster agent inspects pod annotations only, so annotations placed on a parent object (Deployment or StatefulSet) are not picked up.


  • Topology map — see dependencies between services visually.
  • Incidents — the incidents raised by SLO breaches.
  • Distributed tracing — explore all trace data.
  • Log viewer — search all logs together.
  • Deployments — deployment history.
  • Using charts — how to read charts, zoom, overlay and the other shared chart features.
  • Settings — configuring inspection conditions and managing application categories