Skip to content

10.3. Troubleshooting

How to diagnose and resolve problems with installation, agent connections, missing data and performance.

Overview​

Where you run into a problem using OPENMARU Observability, use this document to establish the cause and resolve it. Each kind of problem comes with the order to check things in and what to do.

Note: the kubectl examples in this document use -n openmaru-observ as the namespace. If you installed into a different namespace, substitute its name.

Installation problems​

A pod does not start, or keeps restarting​

Symptom: kubectl get pods -n openmaru-observ shows a pod as Pending, CrashLoopBackOff or Error.

Cause and action:

  1. Check the storage class

    Where PersistentVolume provisioning fails, the pod stays Pending.

    kubectl get pvc -n openmaru-observ

    Where the PVC is Pending, check that the storage class you specified at Helm install time exists on the cluster.

    kubectl get storageclass

    Find the name of an available storage class, set it correctly in values.yaml and reinstall.

    Note: the Helm chart's default storage class is nfs-client. Change it to one that suits your cluster.

  2. Check for insufficient resources

    Where a node has no CPU or memory to spare, the pod is not scheduled.

    kubectl describe pod <pod-name> -n openmaru-observ

    Where you see Insufficient cpu or Insufficient memory, free up cluster resources, or reduce each component's resource requests in values.yaml and reinstall.

  3. Check for an image pull failure

    ImagePullBackOff means the container image registry cannot be reached.

    kubectl describe pod <pod-name> -n openmaru-observ

    Read the image-related error in the Events section. Check the registry address and the network connection. In an air-gapped environment, check that the images are registered correctly in the internal registry.

    Note: where you use a private registry, imagePullSecrets must be set in values.yaml. Check that the Kubernetes Secret was created correctly.

  4. The server pod takes too long to start

    On its first start the server pod performs initialisation such as database migration. This can take up to five minutes, so wait before checking the state.

    Where the server pod stays in CrashLoopBackOff, read its logs.

    kubectl logs <server-pod-name> -n openmaru-observ

    The cause is usually a failure to connect to PostgreSQL or ClickHouse. Check that those pods are running first.

The UI is unreachable​

Symptom: the page does not open when you go to the OPENMARU Observability address in a browser.

Cause and action:

  1. Check the ingress or route

    Check that ingress was enabled at Helm install time.

    kubectl get ingress -n openmaru-observ

    Where there is no ingress resource, enable ingress in values.yaml and reinstall.

    On OpenShift, check the route resource instead.

    kubectl get route -n openmaru-observ

  2. Check the pod states

    kubectl get pods -n openmaru-observ

    Check that the server and UI pods are Running. Where they are not, see A pod does not start, or keeps restarting above.

  3. Check the services

    Check that the services connect to one another correctly inside the cluster.

    kubectl get svc -n openmaru-observ

    The UI pod connects to the server to fetch data. Check that the server service was created correctly.

  4. Check the network

    Check that the browser can reach the server address. A firewall or security group may be blocking it.

A kube-state-metrics warning appears​

Symptom: the project status on the Settings > System settings tab shows kube-state-metrics as unhealthy.

Cause: kube-state-metrics is not installed on the cluster, or cannot be reached.

Action:

kubectl get pods -A | grep kube-state-metrics

Where kube-state-metrics is absent, install it first. It is the component that exposes Kubernetes resource state as metrics, and OPENMARU Observability needs it to build the application list.

Check too that the kube-state-metrics included in the Helm chart is enabled — confirm that component is on in values.yaml.

Where it is installed and the warning still appears, check that VictoriaMetrics is collecting kube-state-metrics' metrics. The kube-state-metrics pod's container name must match the configuration.

A database connection error occurs​

Symptom: the server pod's logs show a database connection failure.

Cause and action:

  1. Check the PostgreSQL pod

    kubectl get pods -n openmaru-observ | grep postgres

    Check that the PostgreSQL pod is Running. On its first start, PostgreSQL can take about two minutes to initialise.

  2. Check the ClickHouse pod

    kubectl get pods -n openmaru-observ | grep clickhouse

    Check that the ClickHouse pod is Running.

  3. Check the storage

    Check that the database pods' PersistentVolumes bound correctly.

    kubectl get pvc -n openmaru-observ

    Check that the PVCs are Bound. Where the storage is full, enlarge the volume or clear out data you no longer need.

  4. Read the database pods' logs

    The problem may be in a database pod itself. Read its logs.

    kubectl logs <postgres-pod-name> -n openmaru-observ
    kubectl logs <clickhouse-pod-name> -n openmaru-observ

Pods are unhealthy after a Helm upgrade​

Symptom: some pods do not start correctly after helm upgrade.

Cause and action:

  1. Check the upgrade status

    helm status openmaru-observ -n openmaru-observ

    Check that the deployment status is deployed.

  2. Check the pod events

    kubectl describe pod <pod-name> -n openmaru-observ

    Read the cause in the Events section.

  3. Check the server pod's ordering

    The server pod needs PostgreSQL and ClickHouse running first. Check that the database pods are all Running, then restart the server pod.

    kubectl rollout restart statefulset <server-statefulset-name> -n openmaru-observ

Agent connection problems​

The node agent is not recognised​

Symptom: the project status on the Settings > System settings tab shows the agent as unhealthy. No nodes appear in the Nodes menu.

Cause and action:

  1. Check the API key

    The node agent authenticates with an API key. Check that a key has been issued on the Settings > System settings tab.

    Check that you used the right API key when installing the agent. If you have issued a new key, the agent must be redeployed.

  2. Check the server URL

    Check that the agent is configured with a URL at which it can reach the OPENMARU Observability server. The node the agent runs on must be able to reach that URL.

  3. Check the agent pods

    Where the node agent is deployed as a DaemonSet, check that its pod is running on each node.

    kubectl get pods -n <the namespace the agent is deployed in> -o wide

    Where a pod is unhealthy, read its logs.

    kubectl logs <pod-name> -n <namespace>

  4. Check the kernel version

    The node agent uses eBPF and needs Linux kernel 4.16 or later. Below that, the agent exits at start-up.

    uname -r

  5. Check the security context

    The node agent must run in privileged mode to load eBPF programs. Check that the cluster's security policy (PodSecurityPolicy, Pod Security Standards and so on) allows privileged mode.

    Check too that the agent pod's volume mounts are in place.

    • /host/sys/fs/cgroup (the cgroup filesystem, read-only)
    • /sys/kernel/tracing (tracefs)
    • /sys/kernel/debug (debugfs)

  6. Wait after installing the agent

    On a first installation, the first data can take one to two minutes to arrive. Wait a moment, then refresh the Settings > System settings tab.

Note: the node agent requires a Linux operating system with kernel 4.16 or later. Below that, eBPF-based collection does not work.

The cluster agent does not connect​

Symptom: cluster-level data such as the dashboard and the application list is missing.

Cause and action:

  1. Check the cluster agent pod.

    kubectl get pods -n openmaru-observ | grep cluster-agent

  2. Where the pod is unhealthy, read its logs.

    kubectl logs <cluster-agent-pod-name> -n openmaru-observ

  3. Check that the following are set correctly in values.yaml.

    ItemWhat to check
    Cluster agent server URLThat the server can be reached at this URL
    Cluster agent API keyThat the API key is valid

  4. The cluster agent needs permission to reach the Kubernetes API. Check that its ClusterRole and ClusterRoleBinding were created correctly.

    kubectl get clusterrole | grep cluster-agent
    kubectl get clusterrolebinding | grep cluster-agent

    The cluster agent needs the following permissions.

    • nodes, services, endpoints, pods, secrets (core API)
    • replicasets, deployments, statefulsets, daemonsets (apps API)
    • jobs, cronjobs (batch API)

OpenTelemetry data does not arrive​

Symptom: the trace or log data from an application instrumented with the OpenTelemetry SDK does not appear in the Distributed tracing or Log viewer menus.

What to check, in order:

  1. Check the OTel Collector pod

    kubectl get pods -n openmaru-observ | grep otel-collector

    Check that the OTel Collector pod is Running.

  2. Check the endpoint configuration

    Check that the endpoint in your application's OpenTelemetry SDK configuration points at the OTel Collector service correctly. Click the OpenTelemetry integration button at the top of the Distributed tracing page to see the correct endpoint.

    The OTel Collector receives data on these ports.

    • gRPC: 4317
    • HTTP: 4318

  3. Check the API key

    The OTel Collector needs a valid API key to send data to the OPENMARU Observability server. Check the API key setting in values.yaml.

  4. Check the network

    Check that the application can reach the OTel Collector, and that the OTel Collector can reach the OPENMARU Observability server.

  5. Read the OTel Collector's logs

    The OTel Collector's logs show what it is receiving and sending.

    kubectl logs <otel-collector-pod-name> -n openmaru-observ

    Where data is being dropped because a memory limit is exceeded, raise the OTel Collector's memory limit in values.yaml.

When data does not appear​

There is no cluster data on the dashboard​

Symptom: the cluster resource gauges and health cards on the dashboard are empty.

What to check, in order:

  1. Check the real-time connection

    Check that the Live badge at the top of the dashboard is green. Grey or an error state means the real-time connection to the server has dropped. Refreshing the page reconnects automatically.

  2. Check the cluster agent

    See The cluster agent does not connect above.

  3. Check kube-state-metrics

    Without kube-state-metrics working, cluster state data cannot be collected. See A kube-state-metrics warning appears above.

  4. Check VictoriaMetrics

    The dashboard's cluster resource data comes from VictoriaMetrics.

    kubectl get pods -n openmaru-observ | grep victoria-metrics

    Check that the VictoriaMetrics pod is running.

The application list is empty​

Symptom: nothing appears in the list when you click the Applications menu.

What to check, in order:

  1. Check the filters

    Check that the filters at the top of the page (namespace, category, status) are not excluding your data. Reset them all and look again.

  2. Check the time range

    Check that the time picker at the top right is set to the last hour or three hours. A range set too far in the past may hold no data.

  3. Check the cluster agent and kube-state-metrics

    Both must be working for the application list to be built.

  4. Check the namespace

    Check that the namespaces you monitor have workloads running. Namespaces without workloads show no applications.

A metric chart has no data​

Symptom: the charts are empty on the metric tabs of an application detail page, or on a node detail page.

What to check, in order:

  1. Check the time range

    There may be no data in the range set in the time picker at the top right. If you have just installed the agent, choose the last hour.

  2. Check the node agent

    Check that the agent is installed and running on that node. The Nodes menu shows the node list and their states.

  3. Check the VictoriaMetrics (metric store) connection

    Check that VictoriaMetrics shows as connected in the Node status section of the Settings > System settings tab. This reflects the connection to the time-series database (VictoriaMetrics) that stores metric data.

    kubectl get pods -n openmaru-observ | grep victoria-metrics

  4. Check VictoriaMetrics' scrape targets

    Check that VictoriaMetrics is collecting the agent's metrics correctly. The node agent pods must carry the metric scrape annotations.

No logs appear​

Symptom: the Log viewer menu, or the Logs tab of an application detail page, returns nothing.

What to check, in order:

  1. Reset the filters

    Check the conditions set in the application filter at the top of the Log viewer and in the attribute filter panel on the left, and clear any you did not intend.

  2. Check the severity filter

    Check that the level you want is selected in the log level filter (ERROR, WARN, INFO, DEBUG).

  3. Check the time range

    Set the time picker at the top right to the last hour and query again.

  4. Check that the node agent is collecting logs

    The node agent must be installed for container logs to be collected. Where the agent was installed with log collection disabled, check its configuration.

  5. Check ClickHouse

    Log data is stored in ClickHouse. Check that the ClickHouse pod is running.

    kubectl get pods -n openmaru-observ | grep clickhouse

    Where ClickHouse's storage is full, log queries can fail. Check the PVC.

  6. Check the log source

    On the Logs tab of an application detail page you can switch the log source between agent and OpenTelemetry (otel). Check the right source is selected.

There is no distributed trace data​

Symptom: the heatmap or the trace list in the Distributed tracing menu is empty.

What to check, in order:

  1. Check eBPF automatic collection

    The node agent collects HTTP and gRPC traffic automatically with eBPF. This works only on servers with kernel 4.16 or later. Check the server's kernel version.

    uname -r

  2. Check whether OpenTelemetry is integrated

    Collecting more detailed trace data requires instrumenting the application with the OpenTelemetry SDK. Click the OpenTelemetry integration button at the top of the Distributed tracing page to see how.

  3. Check the time range and filters

    Reset the time picker and the filter conditions and query again. The distributed tracing screen offers filters on service name, span name, status code and more, so check that a filter is not the reason for the empty result.

  4. Check ClickHouse

    Trace data is stored in ClickHouse. Without ClickHouse working, distributed trace data cannot be queried.

Note: eBPF-based automatic collection works with no application code changes. Adding OpenTelemetry gives you detailed span data at function level.

There is no profiling data​

Symptom: no flame graph appears on the Profiling tab of an application detail page.

What to check, in order:

  1. Check that profiling is enabled

    Profiling is enabled by default. Check the profiling settings on the application detail page.

  2. Check the node agent

    Profiling data is collected by the node agent on each node, with eBPF. Check that the node agent is running. (For a particular application's pprof profiles, the cluster agent scrapes them additionally where the openmaru.io/profile-scrape: "true" annotation is present.)

  3. Check the time range

    Choose the last hour in the time picker and query again.

  4. Check that the runtime is supported

    Profiling works on the supported runtimes (Java, Go, Python, .NET, Node.js and so on). Check that the application you are monitoring runs on one.

No services appear on the topology map​

Symptom: the service connection graph in the Topology map menu is empty.

What to check, in order:

  1. Check the real-time connection

    Check that the Live badge at the top of the Topology map is green.

  2. Check the node agent

    Connection information between services comes from the node agent's eBPF-based TCP connection tracking. Check that the node agent is installed correctly on every node.

  3. Check that there is traffic

    Without traffic between services, no connections appear on the topology map. Check that real requests are reaching your applications.

Performance problems​

The dashboard is slow to load​

Symptom: the dashboard or another screen takes a long time to load.

Cause and action:

  1. Check the server's resources

    Check the CPU and memory usage of the OPENMARU Observability server pod.

    kubectl top pods -n openmaru-observ

    Where usage is high, raise the server's resource requests in values.yaml, or increase the number of server replicas, and reinstall.

  2. Check ClickHouse

    High storage usage in ClickHouse — which stores log, trace and profiling data — degrades query performance.

    kubectl get pvc -n openmaru-observ | grep clickhouse

    Where usage is high, enlarge the ClickHouse volume in values.yaml and reinstall, or shorten the retention period so old data is removed automatically.

  3. Check VictoriaMetrics' storage

    Check the storage of VictoriaMetrics, which stores metric data, too. Insufficient storage degrades metric query performance.

    kubectl get pvc -n openmaru-observ | grep victoria-metrics

  4. Narrow the query time range

    The wider the range you query at once, the more there is to process. Narrowing the range in the time picker improves performance.

Note: choosing a range longer than three days on an application detail page still shows only three days of data (with a "3-day limit" badge), because of the raw metric retention period (three days by default in VictoriaMetrics).

The agent uses too much of the server's resources​

Symptom: CPU or memory usage rises on the nodes where the node agent is installed.

Cause and action:

  1. Widen the collection interval

    Collecting metrics less often reduces the agent's resource use. Set the metric collection interval higher than the default (15s) at Helm install time.

  2. Disable L7 tracing

    In environments with heavy network traffic, eBPF-based L7 tracing can use substantial resources. Where you do not need tracing, disable it in the agent's configuration.

  3. Disable log parsing

    In environments with very high container log volume, disabling log parsing reduces the agent's load.

  4. Set resource limits

    Check that the node agent's resource limits are appropriate. You can adjust its CPU and memory limits in values.yaml to suit your environment. The defaults are a request of 500m CPU and 500Mi memory, and a limit of 4Gi memory (there is no CPU limit).

  5. Check the buffer disk usage

    Where the node agent fails to send data, it buffers it temporarily on local disk. A persistent connection problem therefore raises disk usage. Check the server connection first.

ClickHouse storage grows quickly​

Symptom: the ClickHouse PVC's storage usage grows faster than expected.

Cause and action:

  1. Check the retention periods

    Check the retention (TTL) settings for log, trace and profiling data. Shortening them removes old data automatically.

  2. Check the log volume

    Check whether an application is producing excessive logs. Analysing log patterns in the Log viewer identifies the applications producing unnecessary volume.

  3. Enlarge the storage

    Adjust the volume with openmaruObservClickhouse.persistentVolume.size in values.yaml. The default is 300Gi; environments with a lot of log and trace data need more. See the sizing table in Installation.

A custom dashboard log panel is empty, or shows a warning​

Symptom: a log panel on a custom dashboard has no data, or warns that a field has many distinct values.

Cause and action:

  1. Check that ClickHouse is configured

    The logs and audit log data sources need ClickHouse. The template gallery hides the log templates where it is absent, but the panel editor still offers the sources. If a panel you created stays empty, check this first.

  2. Narrow the grouping key

    Group by a field with very many distinct values (a request ID, say) and a warning appears. The chart would run to thousands of bars, so change the key or filter the range down.

  3. Check the query range

    Where the range is wide, the server folds it to at most 24 hours for the aggregate. Longer periods are not shown.

A custom dashboard realtime widget is not moving​

Symptom: the request viewer or realtime request monitor placed on a custom dashboard looks frozen.

Cause and action:

  1. Check the panel size

    Below 160px wide or 72px tall the animation stops. Try making the panel bigger.

  2. Check for traffic

    Realtime widgets draw the requests happening now. With no traffic in that namespace, empty is correct.

  3. Check the per-dashboard limit

    One dashboard holds up to three distinct filter combinations of realtime widgets. Past that, the data source turns disabled in the panel editor with the reason shown.

VictoriaMetrics is short of storage​

Symptom: the VictoriaMetrics pod restarts, or data is missing from the metric charts.

Cause and action:

  1. Check the PVC's capacity

    kubectl get pvc -n openmaru-observ | grep victoria-metrics

  2. Check the retention period

    VictoriaMetrics retains data for three days by default (--retentionPeriod=3d). Extend the retention period, or enlarge the PVC where storage is short. The default size is 10Gi.

  3. Check what is being collected

    Check whether unnecessary metrics are being collected. An application you do not monitor exposing excessive metrics raises storage usage.

Screen problems​

The live connection does not hold​

Symptom: the Live badge at the top of the dashboard or the topology map turns grey, or keeps trying to reconnect.

Cause and action:

  1. Check whether you clicked through a certificate warning. If you reached the site through Advanced > Proceed without adding your internal certificate authority (CA) certificate to the browser's trust store, that exception expires after seven days. When it does, the live connection drops silently first. See A certificate warning appears below.
  2. Where an ingress or proxy sits between the server and the UI, check that it allows WebSocket connections (ws:// or wss://).
  3. Where the proxy's connection timeout is too short, the WebSocket connection drops often. Raise the timeout.
  4. Check that a firewall is not blocking WebSocket connections.

Where the page shows a Reconnect button, click it to reconnect immediately. Refreshing the page also reconnects.

A certificate warning appears​

Symptom: connecting brings up a "Your connection is not private" warning with the error NET::ERR_CERT_AUTHORITY_INVALID. It can also reappear after a few days of working fine.

Cause: the PC does not have your internal certificate authority (CA) in its trust store. There is nothing wrong with the certificate itself; the PC simply does not know the authority that issued it.

Clicking Advanced > Proceed on the warning does get you in, but that is a temporary exception lasting only seven days. After seven days the warning returns — and before it does, the live connection drops first. Reopening or refreshing the page does not extend the exception.

Action: adding your internal root CA certificate to the PC's trust store stops it recurring. For the detailed procedure on Windows and macOS, see the Trusting the certificate in your browser section of Installation.

Where there are many PCs, Windows group policy can distribute it in bulk. The procedure is in the same section.

Note: where you connect to several clusters, each has its own certificate. Registering it for one environment leaves the warning in place for the others.

Alerts are not sent​

Symptom: no alert arrives in Slack, Teams and so on when an incident occurs.

What to check, in order:

  1. Check the alert channel configuration

    Check that the alert channel is connected on the Settings > Alert channels tab. The channel's connection state should show a check mark.

  2. Check the base URL

    Check that the Base URL field on the Settings > Alert channels tab holds a OPENMARU Observability URL reachable from outside. Without it, the links in alert messages may not be built correctly.

  3. Check the webhook URL is valid

    Check that the Slack or Teams webhook URL is still valid. Where it has changed, set it again on the Settings > Alert channels tab.

  4. Check which events the channel receives

    Check that the incident alerts and deployment alerts options are enabled in the channel's configuration.

  5. Send a test alert

    Send a test alert from the Settings > Alert channels tab to confirm the integration works.

  6. Check the server's network

    Check that the OPENMARU Observability server can reach the external webhook URL. In an air-gapped environment, outbound connections to external webhook services may be blocked.

No incidents are raised​

Symptom: an application is producing errors but no incident appears in the Incidents menu.

What to check, in order:

  1. Check the inspection conditions

    Check that the inspection conditions are enabled on the Settings > Inspection conditions tab. Their thresholds must be set appropriately for incidents to be raised.

  2. Check the resolved toggle

    Where the resolved toggle in the Incidents menu is off, already-resolved incidents are hidden. Turn it on to see them all.

  3. Check the agent

    The node agent must be working for the inspection conditions to be evaluated against metric data. Check the agent's connection.

  4. Check the SLO configuration

    Incidents are raised from SLO breaches. Check that the SLO configuration (the availability and response time targets) is set correctly on the application detail page.

Diagnostic commands​

The commands you will use most often.

Checking every pod​

kubectl get pods -n openmaru-observ

Checking one pod in detail​

kubectl describe pod <pod-name> -n openmaru-observ

Reading a pod's logs​

# Recent logs
kubectl logs <pod-name> -n openmaru-observ

# The previous pod's logs (where it restarted)
kubectl logs <pod-name> -n openmaru-observ --previous

Checking PVCs​

kubectl get pvc -n openmaru-observ

Checking services​

kubectl get svc -n openmaru-observ

Checking resource usage​

kubectl top pods -n openmaru-observ

Checking ingress or routes​

# Kubernetes Ingress
kubectl get ingress -n openmaru-observ

# OpenShift Route
kubectl get route -n openmaru-observ

Checking RBAC resources​

kubectl get clusterrole | grep openmaru
kubectl get clusterrolebinding | grep openmaru

If none of this resolves it​

Where the above does not resolve the problem, gather the following and contact OPENMARU technical support.

  • Your OPENMARU Observability version (from the version information at the bottom left of the screen)
  • Your Kubernetes version (kubectl version)
  • Which menu the problem occurs in, and how to reproduce it
  • The relevant pod logs (kubectl logs <pod-name> -n openmaru-observ)
  • The relevant pod events (kubectl describe pod <pod-name> -n openmaru-observ)