A.1. Reference and Troubleshooting
Overview
This chapter gathers the full list of auto-instrumentation labels and annotations, injected environment variables, and HPA metrics, together with frequently asked questions and the commands used for checking.
1. Label and Annotation Reference (Auto-Instrumentation)
One label selects the instrumentation target; every detail setting is an annotation.
Label -- spec.template.metadata.labels
| Label | Required | Default | Allowed values / example |
|---|---|---|---|
openmaru.io/was-agent | required | — | 'true' |
Annotations -- spec.template.metadata.annotations
| Annotation | Required | Default | Allowed values / example |
|---|---|---|---|
openmaru.io/container-names | optional | all | Comma-separated container names (e.g. testapp, app,sidecar) |
openmaru.io/was-agent-version | optional | set at install time | Image tag (e.g. 5.1.0-11.1) |
openmaru.io/was-agent-image-pull-policy | optional | IfNotPresent | IfNotPresent / Always |
These three are still read from labels for existing deployments. When the same key is present as both a label and an annotation, the annotation wins.
2. Injected Environment Variable Reference
One of the same name that already exists is not overwritten. JAVA_TOOL_OPTIONS alone is appended.
| Environment variable | Default behaviour |
|---|---|
OMAPM_HOST | The Operator's OMAPM_HOST when unset |
OMAPM_PORT | The Operator's OMAPM_PORT when unset |
OMAPM_APPLICATION_NAME | <container>-${HOSTNAME:-:2} |
OMAPM_INSTANCE_ID | <container>-${HOSTNAME:-:2}-${HOSTNAME:-:3} |
OMAPM_TRANSACTION_TRACE_THRESHOLD | 500 (ms) |
JAVA_TOOL_OPTIONS | -javaagent:/khan-agent/khan-agent-<version>.jar (appended) |
CATALINA_OPTS_APPEND, JAVA_OPTS_APPEND and JAVA_OPTS are left alone. Whatever the application
sets stays as it is.
The default of OMAPM_APPLICATION_NAME contains the ReplicaSet hash, so it changes on every
redeploy. If you autoscale on it, set the value yourself to pin it
(302 section 3.1).
3. Operator Installation Environment Variable Reference
| Key | Component | Meaning |
|---|---|---|
IMAGE_REGISTRY | Agent | The registry for the khan-agent image to inject |
IMAGE_NAMESPACE | Agent | The namespace within the registry (leading / included) |
OMAPM_HOST / OMAPM_PORT | Agent | The default APM server for the instrumented target |
Each name/value in envHpa | HPA | The apmAlias → APM server URL mapping |
DEFAULT_AGENT_VERSION | Agent | The default image tag of the agent to inject |
<apmAlias>_ACCESS_KEY | HPA | That APM server's API access key. Pair one with every alias -- without it APM answers HTTP 403 |
TLS_CERT_FILE / TLS_KEY_FILE | both | Paths to the server certificate and key. The chart mounts the secret and fills these in |
4. HPA Metric Reference
| Metric | selector labels | What is queried |
|---|---|---|
tps | apmAlias, groupName | The TPS from {APM}/monitoring/api/metrics/apps/info/{groupName} |
5. Frequently Asked Questions (FAQ)
Q. Do I have to change the application image?
No. Adding the label is enough. The agent is injected by an init container and the original image is untouched.
Q. I already use -javaagent or OMAPM_HOST.
The environment variables you set win (they are not overwritten). JAVA_TOOL_OPTIONS is appended
after the existing value.
Q. What about a pod with several containers?
Name the containers to instrument with the openmaru.io/container-names annotation.
Q. Can it be used alongside a CPU/memory HPA?
Yes. A standard HPA's metrics array can hold Resource (cpu/memory) and External (tps) together.
Q. I have several APM servers.
Register each under its alias (name) in envHpa and select with apmAlias in the HPA (see
302 Autoscaling).
6. Troubleshooting at a Glance
| Area | Symptom | What to do |
|---|---|---|
| Installation | The Operator pods do not come up | Check the state and logs with kubectl get pod -n openmaru-apm, and check Helm enabled: true |
| Instrumentation | The agent is not injected | Check the label position (spec.template.metadata.labels) and the value 'true' |
| Instrumentation | The image fails to pull | Check the image built from IMAGE_REGISTRY/IMAGE_NAMESPACE plus was-agent-version exists |
| Instrumentation | Not shown in the APM console | Check OMAPM_HOST/OMAPM_PORT are reachable |
| HPA | TARGETS <unknown> | kubectl describe hpa <name> shows the reason in Events |
| HPA | Scaling stopped after a redeploy | The group name has most likely changed. Pin OMAPM_APPLICATION_NAME |
| HPA | HTTP 403 | Check that <apmAlias>_ACCESS_KEY is paired with the alias |
| HPA | Scaling on apdex moves the wrong way | Use apdexDeficit |
| HPA | It does not scale | Review minReplicas/maxReplicas and target.value |
| Install | helm stops saying caBundle/tlsCrt/tlsKey are empty | The certificate values have to be supplied (201 section 5) |
7. Commands for Checking
# Operator state
kubectl get pod -n openmaru-apm
kubectl logs deploy/openmaru-operator-apm-agent -n openmaru-apm
kubectl logs deploy/openmaru-operator-apm-hpa -n openmaru-apm
# confirm instrumentation applied
kubectl describe pod <pod> | grep -E "khan-agent-init|khan-data"
kubectl set env pod/<pod> --list | grep -E "OMAPM_|JAVA_TOOL_OPTIONS"
# HPA / external metrics
kubectl get hpa
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/<ns>/tps" | jq .