Skip to content

3.1. Pods

When to Look at This​

  • When a deployed application will not start
  • When a service is slow or unresponsive
  • When you need to see logs
  • When you need to check files or configuration inside a container

Most failure investigations start on this screen.

What a Pod Is​

It is the smallest unit Kubernetes deploys and manages. It is one container or a group of containers, and it corresponds to the process actually running in the cluster.

Containers within a pod share the following.

What is sharedMeaning
NetworkThey use the same IP address and reach each other over localhost
StorageThey can attach the same volume and exchange files
LifetimeThey are created together and disappear together
Node placementThey always run on the same node

Why Pods Rather Than Containers​

It would be simpler if a single container were the smallest unit. The extra layer called a pod exists because some containers must stay together.

ExampleMain containerSidecar container
Log shippingThe applicationA collector that reads log files and ships them
Configuration refreshThe applicationA program that periodically downloads configuration
Communication proxyThe applicationA proxy that handles encryption and authentication on its behalf

Such pairs must run on the same node, see the same files, and be created and removed together. Deployed separately, one could land on a different node and they could not share files.

Most pods hold only one container. This is a mechanism for the cases where several are needed.

Why You Do Not Create Pods Directly​

If you create a pod directly, nobody starts it again when it terminates. The same is true if the node goes down.

So in practice you create a controller such as a Deployment and let the controller create and manage the pods (see 3.2).

Most pods you see on this screen were created by a controller. The random string after the name is the sign of that.

Example nameWhat created it
web-7d4f9c8b6-x8k2mDeployment (ReplicaSet hash + pod hash)
db-0StatefulSet (ordinal)
backup-29579160-abcdeCronJob (run time + hash)

The Pod List​

Go to Workloads > Pods.

Pod list
ColumnDescriptionHow to read it
NameThe pod nameThe random string at the end was added by a controller
NamespaceThe namespace the pod belongs to
RestartsHow many times the container has restartedAnything other than 0 means it terminated at least once
ReadyReady containers / total containers1/2 means one is not ready yet
StatusThe current run stateSee below
CPU · MemoryThe amount currently in useClose to the limit suggests a performance problem
IPThe pod's internal addressIt changes when the pod is recreated
NodeThe node the pod is scheduled onIf only one node's pods have problems, suspect the node
AgeTime elapsed since creationTells you whether it restarted recently

Reading Statuses​

StatusMeaningWhat to check
RunningRunning normallyNothing
SucceededFinished its work and exited normallyNothing (normal for a Job)
PendingWaiting to be scheduledNode resource shortage, whether volumes are ready
ContainerCreatingCreating the containerIf it lingers, check image download or volume attachment
FailedFailed to runLogs and events
CrashLoopBackOffRepeatedly starting and stoppingLogs. Usually a configuration error or a dependency that has not started
ImagePullBackOffCould not fetch the imageImage name, registry credentials
TerminatingBeing deletedIf it lingers, termination handling is blocked

The BackOff in CrashLoopBackOff means the retry interval is being widened. On continued failure, Kubernetes widens the interval to 10 seconds, 20 seconds, 40 seconds, and so on, up to five minutes. It is a mechanism to keep endless restarts from consuming the node.

A restart count that keeps rising means the container is terminating repeatedly. Check the logs first.

Hovering over a status shows the cause. The same content appears in the pod list, the pod detail, and the topology preview.

StatusTooltip content
Pending (scheduling failed)"Pod unschedulable" and the reason (e.g. 0/3 nodes are available: 3 Insufficient memory)
CrashLoopBackOffA note that the application keeps failing to start, the last message (up to two lines), the debug button
CreateContainerError · CreateContainerConfigError"Couldn't create container", the waiting message, common causes (missing ConfigMaps, Secrets, or volume mounts, invalid security contexts)
Others (ImagePullBackOff, etc.)The message Kubernetes recorded

A restarting pod briefly shows Error between restarts. The tooltip shows the same note then. Pods that failed scheduling have no debug button. The container never started, so look for the cause in node selection rules and resources.

The Pod Lifecycle​

A pod goes through several stages between creation and receiving traffic. Knowing where it stopped narrows the cause.

OrderStageIf it stops here
1Scheduling — deciding which node to place it onResource shortage, node selection conditions, taints
2Volume attachmentThe PVC is not ready (see 6.1)
3Image downloadTypo in the image name, registry authentication
4Container startCommand error, missing required environment variable
5Passing the readiness probeThe application is still starting, or the probe path is wrong

Probes​

A container being up and being ready to work are different. The process may have started but still be reading configuration or unable to connect to the database.

Kubernetes cannot tell this by looking inside the container. So it asks the application directly. That way of asking is a probe.

Three Probes​

ProbeWhat it asksIf it fails
readiness"Can you take requests right now?"No traffic is sent. It is removed from the Service endpoints
liveness"Are you alive?"The container is restarted
startup"Has startup finished?"It restarts. Until it finishes, the other two probes are deferred

The readiness probe is the key mechanism for zero-downtime deployment. A new pod takes no traffic until it is ready, so requests do not fail during a deployment (see 5.1).

The startup probe is for applications that take a long time to start. For example, if a Java application that takes three minutes to start has only a liveness probe, it is judged unresponsive while still starting and is restarted over and over. A startup probe defers the liveness probe during that time.

Probe Methods​

MethodHow success is judgedWhere it is used
HTTP GETCalls the given path and succeeds if the response code is 200-399Web applications. The most common
TCP socketSucceeds if it can connect to the given portNon-HTTP servers (databases and so on)
Command executionRuns a command inside the container and succeeds if the exit code is 0When the first two cannot decide

When using HTTP GET, it is better to have a dedicated path. Using a real service path fills the logs with probe requests, and if that path calls the database, the pod also restarts whenever the database slows down.

Tuning Values​

You can see each probe's settings on the pod detail.

ValueMeaningDefault
delayHow long to wait after the container starts before the first probe0 seconds
periodThe probe interval10 seconds
timeoutHow long to wait for a response1 second
successHow many consecutive successes count as success1
failureHow many consecutive failures count as failure3

In the Console these values appear as delay = 10s, period = 10s, failure = 3.

The actual time until action is the combination of these values.

Time to first probe = delay
Time to failure verdict = delay + (period × failure)

For example, with delay = 10s, period = 10s, and failure = 3, a restart is triggered 40 seconds after the container starts.

When Probe Settings Cause Problems​

Probes exist to help, but set wrong they keep killing healthy pods.

SymptomCauseRemedy
Restarts repeat but there are no errors in the logThe liveness probe is too strictIncrease delay and failure, or add a startup probe
A slow-starting app keeps restartingdelay is shorter than the startup timeUse a startup probe
The pod is Running but no traffic arrivesThe readiness probe is failingCheck that the probe path and port are right
It restarts only under heavy loadtimeout is short and slow responses count as failuresIncrease timeout
The app restarts whenever the database slows downThe probe path calls the databaseProvide a separate path for probes

Do not put a database connection check in the liveness probe. If the database slows down briefly, every pod restarts at once and the situation gets worse. Such checks are safer in the readiness probe — traffic pauses briefly but nothing restarts.

Which Probes to Use​

SituationRecommendation
Web serviceA readiness probe is essential. Keep the liveness probe generous
Slow startupAdd a startup probe
Batch work (Job)Usually not needed
You do not know which to useUse only a readiness probe. A liveness probe is optional

Without a liveness probe you cannot revive a hung process, but in most cases the container ends when the process ends and Kubernetes restarts it. No liveness probe is better than a badly set one.

How to Configure It — YAML​

Write each probe type into the container spec.

containers:
- name: app
image: registry.example.com/my-app:1.0
ports:
- name: http
containerPort: 8080
startupProbe: # has startup finished — the two below begin only after this passes
httpGet:
path: /health/startup
port: http
periodSeconds: 10
failureThreshold: 30 # 10 seconds x 30 = waits up to five minutes
readinessProbe: # may it take traffic
httpGet:
path: /health/ready
port: http
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
livenessProbe: # is it alive
httpGet:
path: /health/live
port: http
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 3
FieldIn the table aboveDescription
initialDelaySecondsdelayHow long to wait before the first probe
periodSecondsperiodThe probe interval
timeoutSecondstimeoutHow long to wait for a response
failureThresholdfailureHow many consecutive failures count as failure
successThresholdsuccessHow many consecutive successes count as success

You can use a name in port. Naming the containerPort as in the example above means you do not have to edit probe settings when the port number changes.

With a startup probe you do not need initialDelaySeconds to buy time. It suits applications with variable startup times particularly well — if they start quickly they pass quickly, and if they take longer it waits periodSeconds x failureThreshold.

Non-HTTP servers use other methods.

readinessProbe:
tcpSocket: # only checks whether the port is open
port: 5432
livenessProbe:
exec: # succeeds if the command exit code is 0
command: ["sh", "-c", "pg_isready -U postgres"]

Pod Detail​

Selecting a name in the list opens the detail screen.

Pod detail
ItemWhat it tells you
LabelsThe criteria a Service uses to select this pod (see 5.1)
AnnotationsInformation left by deployment tools, such as the last restart time
OwnerThe parent resource that created this pod. Select it to trace back
StatusThe current run state
NodeThe node it is scheduled on. Select it to check node status
Service accountThe account this pod uses to call the cluster API (see 7.1)
Host IP · Pod IPThe node address and the pod address
QoS classThe order in which it is evicted when resources run short

Scrolling down shows per-container information.

ItemDescription
ImageThe running container image and tag
StatusPer-container run state and the last termination reason
Environment variablesInjected configuration values. Those from ConfigMaps and Secrets are shown too
MountsAttached volumes and paths
Requests and limitsThe requested CPU and memory, and the limits

Pay attention to the last termination reason. OOMKilled means it was killed for exceeding the memory limit; Error means the application ended with an error.

Resource Requests and Limits​

Each container can set two values: a request and a limit. The names are similar, but they apply at different times and to different things.

ValueMeaningWhen it applies
RequestThis much must be guaranteedWhen deciding which node to schedule the pod on
LimitIt cannot use more than thisWhile the pod is running

Requests Reserve a Place​

When scheduling a pod, Kubernetes adds up the requests to calculate the room left on a node. It does not look at actual usage.

If a node with 8 CPU cores holds three pods requesting 2 cores each, then even if those pods actually use only 0.1 core, the node is calculated as having only 2 cores left. A pod requesting 4 cores is not scheduled and waits in Pending.

Conversely, setting no request reserves no place. Scheduling becomes easier, but with no guaranteed share the pod is evicted first when the node gets crowded (see QoS classes below).

Limits Restrain at Runtime​

What happens when a limit is exceeded differs between CPU and memory.

ResourceWhen the limit is exceeded
CPUIt slows down. It is restricted (throttled) to the limit but is not killed
MemoryIt is killed. The container ends as OOMKilled and starts again

CPU can simply be shared for a while, whereas memory already taken cannot be reclaimed. If OOMKilled repeats, raise the limit or reduce the application's memory use.

Reading the Values​

CPU is written in cores and memory in byte sizes.

NotationMeaning
2 or 2000m2 CPU cores. m is 1/1000, so 500m is 0.5 core
1.5Gi1.5 gigabytes of memory
512Mi512 megabytes of memory

What Happens If You Set Only One​

What you setResult
Request onlyWith no limit, it uses as much as the node has spare
Limit onlyThe request is filled in automatically with the same value as the limit
BothExactly as set
NeitherIt reserves no place and has no restriction

The important point is that setting only a limit still creates a request. The application charts COP provides are like this — they set only limits of cpu: "2" and memory: 1.5Gi, so the requests become the same values. Such a pod occupies 2 cores on the node regardless of actual usage.

In the Console, check Requests and limits under the container section of the pod detail. If the two values are the same, either only the limit was set or they were deliberately made equal.

What Values to Use​

If too smallIf too large
RequestIt is evicted when the node gets crowdedFew pods fit on a node and resources sit idle
LimitCPU slows down and memory is killedOne pod uses up the node and affects other pods

Measure actual usage first, then decide. The Console's pod list shows current CPU and memory usage, and the monitoring screen shows change over time (see 9.2). Taking typical usage as the request and the peak as the limit is a starting point.

Java applications have to be set together with the heap size. If the heap is 512MB and the memory limit is also 512MB, the areas outside the heap (metaspace, thread stacks) push it over the limit and it is killed. Set the limit comfortably above the heap.

QoS Classes​

This is the grade that decides which pods are pushed out first when resources run short. You do not specify it; it is determined automatically from the resource settings.

ClassConditionOn resource shortage
GuaranteedEvery container's request equals its limitEvicted last
BurstableOnly requests are set, or requests differ from limitsIn between
BestEffortNeither requests nor limitsEvicted first

If an important application is BestEffort, it is terminated first when resources run short. Specify requests.

Guaranteed is the most stable but holds resources equal to the requests at all times, which lowers cluster density. Burstable suits most cases.

Finding the Cause Through Events​

The events at the bottom of the detail screen tell you the cause of a problem fastest. When a pod will not start, look here before the logs.

Logs exist only after the container starts, whereas events also record the pre-start stages (scheduling failure, image download failure, volume attachment failure).

ReasonMeaningRemedy
FailedSchedulingCould not find a node to place it onCheck node resources, node selection conditions, taints
Failed / ErrImagePullCould not fetch the imageCheck the image name and registry credentials
UnhealthyA probe failedCheck that the application responds on the probe path
BackOffThe retry interval was widened after repeated failuresResolve the root cause first
FailedMountCould not attach a volumeCheck the PVC status (see 6.1)
EvictedPushed out due to node resource shortageCheck the QoS class and node headroom

Viewing Logs​

Select the logs button at the top right of the detail screen.

FeatureDescriptionWhen to use it
Container selectionChooses which container to view when the pod has severalPods with sidecar containers
Auto-refreshFollows the screen down as new logs appearWhen watching what is happening now
Line countSets how many lines to fetchReduce it when many logs make it slow
Previous containerThe log of the container before it restartedPods that restart repeatedly
TimestampsAdds a time to each lineWhen aligning with other records
DownloadSaves the log to a fileWhen sharing or searching

If a pod restarts repeatedly, the current log is often empty and the cause is left in the previous container log.

Logs contain only the container's standard output. If the application writes only to files, nothing appears here. Logs of deleted pods cannot be viewed, so use the monitoring screen if you need them (see 9.2).

The Web Terminal​

Selecting the terminal button lets you run commands inside the container.

It is used in situations like these.

What to checkHow
Whether the configuration file is attached properlyLook at the file at the mount path
Whether it connects to another serviceCall it with curl or nc
Whether environment variables arrivedPrint the environment variables
Whether the process is runningLook at the process list

There are cautions.

  • You cannot connect if the container has no shell. Minimal images often have none.
  • Files created here disappear when the pod restarts.
  • Editing configuration directly in a running pod is reverted on the next deployment. Use it only for temporary checks, and make real changes through a ConfigMap or the deployment definition (see 3.4).

The Debug Container​

You cannot connect with the web terminal to a pod whose app stops right after it starts. In this case, use the Debug container.

The debug container creates a temporary pod (the debug pod) that copies the problem pod. The debug pod runs a shell (/bin/sh) instead of the app. The image, environment variables, ConfigMaps, Secrets, PVCs, and service account are the same as the original pod. You can therefore check how the configuration files actually arrived and whether the connection targets are reachable, under the same conditions.

When the Button Appears​

The debug button (Debug container) appears only for containers in the following states.

Container stateMeaning
CrashLoopBackOffIt starts and stops repeatedly and is waiting to restart
Terminated with an errorIt ended with an exit code other than 0
Restarted after an errorIt is running now, but the previous run ended with an exit code other than 0
  • Containers that ended normally (exit code 0) have no button.
  • The debug pod itself and init containers have no button either.
  • The button appears only when you have permission to create pods and to use the terminal (exec).
Debug button in the pod status tooltip

How to Use It​

  1. Start debugging from one of these two places.
    • The tooltip that appears when you hover over the status in the pod list. Each problem container has a Debug container <container name> button.
    • The log window. When you select a problem container, a Debug container link appears next to the container selection.
  2. If the original pod has PVCs, choose how to connect them in a small window and select Start. If there are no PVCs, this step does not appear.
  3. Wait for the debug pod to start. The window opens at 80% of the screen height, and the progress appears where the terminal will be. Use the full-screen button at the top right to fill the screen.
  4. When the shell connects, run the Original start command shown at the top of the window in the shell to reproduce the error.
  5. To look at another container, change it in the container list above the terminal. The list does not appear when there is only one container.
  6. When you are done, select the close (X) button at the top right of the window, or press Esc.
Debug window

Closing the window deletes the debug pod. If you move to another menu while the window is open, the window closes and the debug pod is deleted too. Clicking outside the window does not close it. In the pod list, a debug pod has a Debug label next to its name.

Closing the browser tab or reloading the page also sends the request to delete the debug pod.

PVC Connection Modes​

ChoiceBehaviorCaution
Read-only (default)Connects the PVCs read-only. You can only view the dataIf an init container writes to a PVC, the debug pod may not start
Read-writeConnects the PVCs the same way as the original podChanges made in the shell apply to the production data as is
Do not connectMounts an empty directory at the same path instead of the PVCThe original data is not visible
Choosing the PVC connection mode

The chosen mode stays visible above the terminal as PVC connection: <mode>. If you close the window at the step where you choose the mode, no debug pod is created.

When the Debug Pod Remains or Does Not Start​

SituationResult
The browser is forcibly closed or the network dropsThe delete request does not arrive. The debug pod terminates automatically after at most 1 hour, and it is deleted the next time you open a debug window in the same namespace
The delete fails because of a network errorThe window shows a failure notice. The console tries the delete again the next time you open a debug window in the same browser page
You have no permission to delete podsThe debug pod remains for 1 hour. The window shows this
It does not start within 2 minutesRecent warning events are shown. Choose Wait again or Close
The image has no /bin/shThe debug pod cannot be used. This image has no /bin/sh. The debug terminal cannot be used. is shown
You run exit in the shellThe window does not close. Press Enter to connect again

Differences from the Original Pod​

SituationResult
The pod has init containersThe init containers run again each time you debug. Take care if they change data (such as migrations). The window shows this at the top
An init container is the cause of the failureThe debug pod also fails at the init stage
Sidecars (Istio proxy and so on)Sidecars also become shells. Networking that goes through a sidecar does not work
A NetworkPolicy that selects targets by labelThe debug pod does not have the original labels, so different rules apply
A pod using an RWO (ReadWriteOnce) PVC is placed on a different nodeIt waits with a Multi-Attach warning. After 2 minutes, the warning events show the cause
RWOP (ReadWriteOncePod) PVCThe debug pod cannot start while the original pod exists
CPU, memory, and GPU requestsIt requests the same amounts as the original pod. It counts against the namespace quota, and it waits if no GPU is free

Common Problems and Remedies​

SymptomWhere to look firstCommon cause
Stuck in PendingEventsNode resource shortage, PVC not ready
CrashLoopBackOffPrevious container log. If the cause is not visible, open a shell with the debug container and run the original start commandConfiguration error, missing required environment variable, dependency not started
ImagePullBackOffEventsTypo in the image tag, missing registry credentials
Running but unresponsiveProbe settings, logsThe application is still getting ready, port mismatch
Restarts repeat with no errors in the logLiveness probe settingsThe probe is too strict
It keeps getting evictedQoS class, node resourcesRequests not set (BestEffort)
OOMKilledContainer status, memory limitThe limit is below actual usage