Skip to content

6. Troubleshooting Guide

This gathers the symptoms met most often after installation and what to do about them. Find the item closest to what you are seeing.

Finding It by Symptom​

SymptomItem
The WAS does not start and a VerifyError appearsAn error occurs while running the WAS
The JVM suddenly dies in productionAn error occurs while running the WAS and JVM core dump
Korean is corrupted on the console's transaction screenText is displayed corrupted
Transaction times do not match realityServer time difference
CPU time per method shows 0Checking the transaction CPU time display
Apache information does not appear in the consoleApache web server information
A startup error occurs on containers or OpenShiftContainer and OpenShift environments

If none of the items fits, gather the following three and tell the technical support team -- the agent version, the Java version, and the full error message from the WAS startup log.

The Korean Text in Transaction Trace Information Is Displayed Corrupted​

OPENMARU APM uses the UTF-8 charset by default. If the WAS uses EUC-KR as its default charset, change the agent configuration as below to convert EUC-KR into UTF-8 so the text prints correctly.

$ vi khan-agent-test.conf
----------
user.charset.encoding=EUC-KR

There Is a Large Server Time Difference between the APM Server and the WAS Machine​

Setting the option below in the OPENMARU APM instance startup script ignores the server time of the WAS/system machine and uses the OPENMARU APM server's time for everything.

-Duse.monitoring.server.time=true

When installing with the provisioning feature of OPENMARU APM, selecting the 'Use monitoring server time (ignore agent time)' option below during installation sets -Duse.monioring.server.time=true.

Checking the Transaction CPU Time Display​

OPENMARU APM measures and displays the CPU time used per method within a transaction. That feature is supported from Java 1.6 onwards, so with the OPENMARU APM agent for Java 1.5 the value shows as 0.

An Error Occurs While Running the WAS​

The error message

26-Feb-2016 16:57:58.334 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[HTTP/1.1-8090]]
  org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-8090]]
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
    at org.apache.catalina.core.StandardService.initInternal(StandardService.java:567)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:851)
      … …
      at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)
Caused by: java.lang.VerifyError: Expecting a stackmap frame at branch target 37
Exception Details:
  Location:

org/apache/catalina/connector/CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V @2: invokestatic
  Reason:
    Expected stackmap frame at this location.
  Bytecode:
    0x0000000: 014e b805 0713 0509 1305 0a13 0522 1305
    0x0000010: 232a 05bd 00f4 5903 2b53 5904 2c53 b605
    0x0000020: 124e a700 06b6 0515 2b04 b600 03c0 0004
    0x0000030: 3a04 2c04 b600 05c0 0006 3a05 1904 c700

Cause

From Java 7 onwards, the JVM checks the stack map frames (StackMapTable) when loading a class. When the agent inserts measurement code into the bytecode, branches and try-catch blocks appear; if the stack map frames are not recalculated at that point they no longer match the original values and the verifier refuses to load the class.

Resolution

Upgrade the agent to the latest version. It has been changed to recalculate the stack map frames of the classes it instruments, so this error does not occur.

Do not use -noverify or -Xverify:none

In the past, -noverify or -Xverify:none was added to the startup options to turn the verifier off and avoid this error. It is not needed with the current agent, and adding it is actively dangerous.

Turning the verifier off meansBad bytecode is not filtered out and is loaded as it is
The result isThe JVM can terminate abnormally (SIGSEGV) at the moment the JIT compiler optimizes that code
When it shows upNot right after loading, but days later, once that code has run enough

There has been a real failure where the JVM terminated in production with this option set. It is a form that is hard to trace, so if the option is still in the startup script, remove it.

From Java 13 onwards, -Xverify:none and -noverify are deprecated and print a warning at startup. Oracle has said it intends to remove them.

If the error persists

Tell the technical support team the agent version together with the class name in the error message.

The Apache Web Server Information Is Not Displayed​

Cause and resolution

The Apache server information is fetched when the system agent starts. If Apache is not running at that point, the information cannot be collected.

Restarting only the system agent while Apache is running collects the server information.

A JVM Core Dump Occurs When Running on a UNIX Server​

On UNIX systems, an error can occur when OPENMARU APM uses Snappy, the speed-efficient compression algorithm, to send data.

Change the compression algorithm to lzw in the WAS agent and SYS agent options as follows.

# agent's compression algorithm : snappy(default), lzw
agent.compress.type=lzw

Container and OpenShift Environments​

LogManager Error with the OpenShift JBoss EAP Image​

Running the JBoss EAP + OPENMARU APM image produces a LogManager error on JBoss EAP 6.4/7.0, as in the following error message.

java.lang.RuntimeException: JBAS014670: Failed initializing module org.jboss.as.logging
at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java
:111)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489)
at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:299)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:292)
at org.jboss.as.server.ServerService.boot(ServerService.java:346)
at org.jboss.as.server.ServerService.boot(ServerService.java:321)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:254)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: JBAS011592: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:103)
... 11 more

Cause and resolution

With JBoss EAP, the location of the LogManager changes with each patch, so change the following part of the contrib/openShift-launch.sh file to match the JBoss EAP container image version in use.

$ cd jboss-eap64/contrib
$ vi OpenShift-launch.sh
… omitted …
export JBOSS_LOGMANAGER_DIR="/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-6.4.14.CP/org/jboss/logmanager/main"
#export JBOSS_LOGMANAGER_DIR="/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-6.4.13.CP/org/jboss/logmanager/main"
#export JBOSS_LOGMANAGER_DIR="/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-6.4.12.CP/org/jboss/logmanager/main"
#export JBOSS_LOGMANAGER_DIR="/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-6.4.11.CP/org/jboss/logmanager/main"
export JBOSS_LOGMANAGER_JAR=`cd "$JBOSS_LOGMANAGER_DIR" && ls -1 *.jar'
… omitted …

Docker CPU Usage Is Not Monitored on OpenShift 3.6​

The following message appears in the OPENMARU APM SYS agent log.

2017-11-13 09:46:06 [INFO ] CpuStat.java::parseCpuStat:90 - FileNotFound file: /sys/fs/cgroup/cpu/system.slice/docker-5561112f9d4762568135d5c91fe2ba610d1cb331cbe86d430c695b4af677100d.scope/cpuacct.stat
2017-11-13 09:46:06 [WARN ] MemStat.java::parseMemory:97 - FoundNotFound file: /sys/fs/cgroup/memory/system.slice/docker-bb16cdd2dd3a674d8eee018ade3fd517077ff718ab67d10fa87678a536c0507f.scope/memory.usage_in_bytes

Cause and resolution Up to OpenShift 3.5, Docker used cgroup as its cgroup driver, but OpenShift 3.6 changed the default Docker cgroup driver to systemd. The location of the pseudo files created for Docker monitoring changed as a result, causing the error.

  • Edit the Docker plugin configuration file
$ cd khan-sys-agent/plugins/khan-plugin-docker.conf
$ vi config.properties
  • Change docker.cgroup.driver.type to systemd
# docker's cgroup driver type : cgroup or systemd

# OpenShift 3.6 use systemd type, below OpenShift 3.5 use cgroup type

docker.cgroup.driver.type=systemd

#docker.cgroup.driver.type=cgroup