4.8. Deployment Strategies

When to Read This Chapter

When deploying a new version and the service must not be interrupted
When you have to revert immediately if a problem appears
When you want to show a new version to some users first
When you have to decide which method to use

Chapters 4.2 through 4.7 covered what you deploy with (Console, Jenkins, Bastion, CI tools, ArgoCD); this chapter covers in what order you switch over. Whichever route you deploy through, you can use the method chosen here.

Comparing the Four

Rolling update and Recreate are settled by Deployment settings, but blue-green and canary are not Deployment settings. They are built out of how you arrange Services.

Strategy	What it is built from	Service interruption	Rollback speed	Resources needed
Rolling update	Deployment settings	None	Rollback takes time	A little more
Recreate	Deployment settings	Yes	Requires deploying again	The same
Blue-green	Two Deployments + switching the Service selector	None	Immediate	Double
Canary	Two Deployments + a replica ratio	None	Fast	A little more

Kubernetes supports only the first two directly. The last two are assembled from features that already exist (a Service's label selection).

Rolling Update — Replacing One at a Time

This is the default. It starts one new pod and, once it is ready, takes one old pod down, and so on. The configuration is settled in one place, the Deployment's strategy (see 3.2).

Advantages	Drawbacks
The configuration is simple	Two versions run at once during the replacement
It uses only a little extra resource	Reverting means another rolling change, which takes time
The service is not interrupted	If you notice a problem late, most pods are already replaced

This is enough in most cases. The methods below are for when circumstances prevent using it.

Recreate — Taking Everything Down and Starting Again

It takes all old pods down and then starts the new ones. The service stops in between.

Use it only when two versions must not run at once.

When to use it	Reason
The database schema changed	The old version cannot read the new schema
They use the same file with `ReadWriteOnce`	Two pods cannot attach at once (see 6.1)
A program that uses an exclusive lock	The two conflict if they run together

Blue-Green — Bringing Everything Up, Then Switching at Once

This is the approach of splitting the same application into two Deployments and changing which one the Service looks at. A Service selects pods by label (see 5.1), so changing that condition alone moves all the traffic.

COP prepares this arrangement for the sample application in advance. Looking at the egov namespace, there are four Deployments.

Deployment	Label	Purpose
`egov`	`application: egov`	The default deployment
`egov-blue`	`application: egov-blue`	One side of blue-green
`egov-green`	`application: egov-green`	The other side of blue-green
`egov-hpa`	`application: egov-hpa`	For demonstrating autoscaling (see 3.5)

The explanation below follows this actual arrangement.

1. Create two Deployments. They are distinguished by a single label.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: egov-blue
  namespace: egov
spec:
  replicas: 2
  selector:
    matchLabels:
      application: egov-blue      # the Service selects on this value
  template:
    metadata:
      labels:
        application: egov-blue
    spec:
      containers:
        - name: egov-blue
          image: registry.example.com/apps/egov:1.0.0

egov-green is the same, with the name and label changed to egov-green and the image set to the new version.

COP also splits the script directories so each side can be built and deployed separately. Under workspaces/apps/egov/ on the Bastion there are separate egov-blue and egov-green directories, each with its own Deployment name in its env.sh (see 4.5). That is how one side alone can be deployed with a new version.

2. The Service points at only one side.

apiVersion: v1
kind: Service
metadata:
  name: egov-blue-green
  namespace: egov
  labels:
    application: egov-blue-green
spec:
  type: ClusterIP
  selector:
    application: egov-blue        # currently sends to blue
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

The Ingress points at this one Service, so the external address does not change when you switch over.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egov-blue-green
  namespace: egov
spec:
  ingressClassName: default
  rules:
    - host: egov-blue-green.apps.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: egov-blue-green    # the Service stays; only what is behind it changes
                port:
                  number: 8080

3. Bring up the new version first and check it. Because no Service points at egov-green, it receives no user traffic. Adding one more Service for checking lets you test it in advance.

apiVersion: v1
kind: Service
metadata:
  name: egov-preview            # for checking only
  namespace: egov
spec:
  selector:
    application: egov-green
  ports:
    - port: 8080
      targetPort: 8080

4. Switch over. Change the Service's selector to egov-green. Edit it on the Service detail in the Console or in the YAML editor (see 1.2).

spec:
  selector:
    application: egov-green      # egov-blue to egov-green

To revert, change this value back to egov-blue and you are done. The old version's pods are still up, so it returns to the original state within seconds. This is the greatest advantage of blue-green.

A Jenkins Job performs this switch for you (see 4.4). Choosing the switch target performs the Service selector change above. That one thing is all it actually does.

Advantages	Drawbacks
Reverting is immediate	Both versions are up at once, so it needs twice the resources
You can check the new version in advance	If they share a database, both versions have to work with it
The two versions do not run mixed together	There are more resources to manage

Do not delete the old version right after switching. A problem can surface late, so it is safer to leave it for a day or two before cleaning up. If the resources matter, reduce its replicas to 0.

Canary — Showing It to Some Users First

You bring the new version up on only a few pods to show it to some users first, and increase the ratio if no problems appear. The name comes from the bird miners took along to detect harmful gas.

COP does not prepare this in advance. Unlike blue-green, there is no ready-made arrangement, so you build it yourself. The resource layout is nearly the same as blue-green, differing only in that the Service points at both sides.

1. Give the two Deployments one common label.

# stable version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: egov-stable
  namespace: egov
spec:
  replicas: 9
  selector:
    matchLabels:
      application: egov-canary   # common — the Service looks at this
      track: stable              # for distinguishing
  template:
    metadata:
      labels:
        application: egov-canary
        track: stable
    spec:
      containers:
        - name: egov
          image: registry.example.com/apps/egov:1.0.0

# new version — only one at first
apiVersion: apps/v1
kind: Deployment
metadata:
  name: egov-canary
  namespace: egov
spec:
  replicas: 1
  selector:
    matchLabels:
      application: egov-canary   # the same value
      track: canary              # only this differs
  template:
    metadata:
      labels:
        application: egov-canary
        track: canary
    spec:
      containers:
        - name: egov
          image: registry.example.com/apps/egov:1.1.0

2. The Service looks only at the common label.

apiVersion: v1
kind: Service
metadata:
  name: egov-canary
  namespace: egov
spec:
  selector:
    application: egov-canary     # does not include track
  ports:
    - port: 8080
      targetPort: 8080

Leaving out track is the key. Including it targets only one side, which is the same as blue-green.

3. Raise the stages by replica ratio.

Stage	`egov-stable`	`egov-canary`	Share the new version receives
Start	9	1	10%
First increase	7	3	30%
Second increase	5	5	50%
Complete	0	10	100%

Raise the stages with the replica count control in the Console (see "Adjusting the Replica Count" above). If you see a problem, taking the canary side to 0 stops it immediately.

Advantages	Drawbacks
Even with a problem, only some users experience it	The ratio is decided only by pod count
You can check with real traffic	Both versions run at once, so they must be compatible
It uses only a little extra resource	A person has to raise the stages

The ratio is not precise. In this method the traffic ratio is the pod count ratio. Splitting finely, such as "only 5%", requires at least 20 pods.

Canary — Splitting by Request Ratio (HAProxy)

In the previous method the ratio is decided by pod count alone. To split precisely by request count, use the route-acl feature of the HAProxy Ingress Controller that COP uses (see 5.1).

Attaching a condition to a Service sends only requests matching that condition to it. The number of pods does not matter.

The Arrangement

1. Keep two separate Services. Unlike the previous section, each points at its own pods only.

apiVersion: v1
kind: Service
metadata:
  name: egov-stable
  namespace: egov
spec:
  selector:
    application: egov
    track: stable
  ports:
    - port: 8080
      targetPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: egov-canary
  namespace: egov
  annotations:
    haproxy.org/route-acl: rand(100) lt 20    # only 20% of requests come here
spec:
  selector:
    application: egov
    track: canary
  ports:
    - port: 8080
      targetPort: 8080

rand(100) lt 20 means draw a number from 0 to 99 per request and be true if it is under 20. Changing the number changes the ratio — lt 5 is 5% and lt 50 is 50%.

2. Create two Ingresses. Give them the same host and the same path.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: egov-canary
  namespace: egov
spec:
  ingressClassName: default
  rules:
    - host: egov.apps.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: egov-canary      # the side with the condition
                port:
                  number: 8080

The stable version's Ingress is the same content pointing at egov-stable.

An Ingress is still required even with route-acl attached. The annotation alone does nothing.

How It Works

The rules the controller generates look like this, and the rule with the condition is placed ahead of the general rule.

use_backend egov_svc_egov-canary_8080 if { host matches } { path matches } { rand(100) lt 20 }
use_backend <general routing>

Incoming requests are checked from the top. 20% match the first line and go to the canary; the remaining 80% fall through and go to the stable version.

Raising the Ratio

Change only the annotation value on the Service. The pod counts are left alone.

Stage	Annotation value	Share the canary receives
Start	`rand(100) lt 5`	5%
First	`rand(100) lt 20`	20%
Second	`rand(100) lt 50`	50%
Complete	Delete the canary Ingress and deploy the stable version with the new image	—

To stop, delete the canary Ingress. The condition disappears and all requests go to the stable version.

Conditions Other Than a Ratio

route-acl takes HAProxy condition expressions as they are. Instead of a random ratio, you can show the new version only to particular targets.

Purpose	Condition expression
N% of requests	`rand(100) lt 20`
Users with a particular cookie	`cookie(canary) -m found`
Requests sending a particular header	`req.hdr(X-Canary) -m str yes`
Only from an internal address range	`src 10.0.0.0/8`

Cookie or header conditions are better for internal validation. With a random ratio the same user can get a different version on each request and the screen flickers between them, whereas a cookie condition keeps that user on the new version.

Comparing the Two Methods

	Replica ratio	`route-acl`
Granularity	Decided only by pod count	Decided freely per request
To reach 5%	You need 20 pods	Just set the number to `lt 5`
How to adjust	Change the replica count	Change the annotation value
The same user	Can differ from request to request	Can be pinned by condition (cookie, header)
Resources needed	Pods in proportion to the ratio	One canary pod is enough
Where it applies	Any environment	Environments using the HAProxy Ingress Controller

Do not put several paths in one Ingress. In an Ingress where several paths can match the same request, this feature does not behave as intended. Keep one path only in the canary Ingress.

The ratio is random. With few requests, the actual distribution differs from the value you set. Over about 100 requests, 20% will not be exactly 20.

What to Choose

Situation	Recommendation
No special circumstances	Rolling update
Two versions must not run at once	Recreate
You must be able to revert immediately if a problem appears	Blue-green
You have twice the resource headroom	Blue-green
You want to check with real traffic first	Canary
You have little resource headroom	Rolling update or canary

Every method requires a readiness probe (see 3.1). If it cannot tell whether new pods are ready, the rolling update stalls, and blue-green and canary send traffic to pods that have not started.

Blue-green and canary require the old and new versions to work with the same data. They cannot be used for deployments that change the schema — for those, use Recreate, or change the schema in stages so that both versions can work with it.

When to Read This Chapter​

Comparing the Four​

Rolling Update — Replacing One at a Time​

Recreate — Taking Everything Down and Starting Again​

Blue-Green — Bringing Everything Up, Then Switching at Once​

Canary — Showing It to Some Users First​

Canary — Splitting by Request Ratio (HAProxy)​

The Arrangement​

How It Works​

Raising the Ratio​

Conditions Other Than a Ratio​

Comparing the Two Methods​

What to Choose​