3.2. Autoscaling (APM HPA)

Overview

Pods are scaled out and in automatically on an APM application metric (TPS). It lets you scale to the real traffic, which CPU and memory alone find hard to catch.

1. How It Works

APM HPA implements the Kubernetes External Metrics API (external.metrics.k8s.io/v1beta1). When the HPA asks for an external metric, APM HPA queries the OPENMARU APM server for that group's metric and returns the value.

• The metric supported is tps (the application group's requests handled per second). That is the external metric provided today.
• groupName: the name of the application group in OPENMARU APM.
• apmAlias: the alias that picks which APM server to ask (the name registered in envHpa at installation).

2. Using It — Creating an HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tomcat-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tomcat
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: External
      external:
        metric:
          name: tps                       # the metric name (tps is supported today)
          selector:
            matchLabels:
              apmAlias: APM-SERVER        # which APM server (matches the name in envHpa)
              groupName: TOMCAT           # the APM application group name
        target:
          type: AverageValue
          value: 50                       # scale towards an average of 50 TPS per pod

Applying it:

kubectl apply -f tomcat-hpa.yaml
kubectl get hpa tomcat-hpa

NAME         REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
tomcat-hpa   Deployment/tomcat   0/50      1         3         1          8s

• The integration is working once the left-hand value of TARGETS (the current TPS) shows properly.

3. Confirming It Works (a Load Test)

# generate load
ab -n 18000 -c 10 http://tomcat-openmaru-test1.apps.example.local/

# pods grow once TPS passes the target
kubectl get hpa tomcat-hpa
NAME         REFERENCE           TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
tomcat-hpa   Deployment/tomcat   506/50     1         3         3          24m

The external metrics API can also be checked directly.

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/<namespace>/tps" | jq .

4. Using Several APM Servers

With several APM servers, register each under a different name (= apmAlias) in envHpa at installation.

  envHpa:
    - name: APM-INTERNAL
      value: "http://openmaru-apm-server.openmaru-apm.svc.cluster.local:8080"
    - name: APM-SERVER
      value: "http://192.168.80.190"
    - name: APM-SERVER_ACCESS_KEY      # that APM server's API access key (<apmAlias>_ACCESS_KEY)
      value: "<APM API access key>"

Then pick between them with matchLabels.apmAlias in the HPA.

            matchLabels:
              apmAlias: APM-INTERNAL    # query the internal APM server
              groupName: TOMCAT

If apmAlias matches no name in envHpa, the query goes to the default server -- match the name exactly to use the server you intended.

5. The Key Settings at a Glance

HPA field	Value	Meaning
metrics[].type	External	Scale on an external metric
metric.name	tps	The metric to use (tps today)
selector.matchLabels.apmAlias	e.g. APM-SERVER	The alias of the APM server to query (the envHpa name)
selector.matchLabels.groupName	e.g. TOMCAT	The APM application group
target.type / target.value	AverageValue / 50	The target TPS per pod

6. When It Does Not Work

Symptom	What to check
TARGETS shows <unknown>	Whether groupName matches the real group name in APM, and whether the apmAlias server address is reachable
It always shows 0	Whether that group has real traffic, and whether the APM server is aggregating the group TPS
It queries a different APM server	Whether apmAlias matches the name in envHpa exactly (case included)
No scaling happens	The minReplicas/maxReplicas range, and whether target.value is too large

---