Skip to content

2.1. Cluster Overview

When to Look at This​

  • To check at a glance in the morning that the cluster is healthy
  • To see whether resources are left when a pod is not being scheduled
  • To find out what is happening, through events

The Overview Dashboard​

Select Cluster in the left menu.

Cluster overview

There are four resource usage tiles at the top.

TileWhat it shows
CPU UsageCPU actually in use, the sum of requests, total capacity
Memory UsageMemory actually in use, the sum of requests, total capacity
PodThe number of running pods and the maximum that can be scheduled
NodesThe number of healthy nodes and the total node count

Reading Resource Usage​

Each tile shows three numbers. They mean different things.

LabelMeaningWhat it tells you
UsedThe amount actually in use nowPerformance problems
AllocatedThe sum of what the pods requestedWhether a new pod can be scheduled
CapacityThe total size of the clusterWhen to add capacity

When a pod will not move past Pending, look at Allocated, not Used. Even if actual usage is 20%, nothing more can be scheduled once Allocated reaches 100%, because Kubernetes schedules based on requests.

Conversely, if usage is 90% while Allocated is 40%, the pods are using far more than they requested. Raising the requests to match reality is more stable.

The Event List​

Below the tiles, recent activity in the cluster is shown.

ColumnDescription
TypeThe kind of resource the event occurred on
NameThe resource name. Selecting it goes to the detail screen
NamespaceThe namespace the resource belongs to
ReasonWhy the event occurred
MessageThe details
Last seenThe most recent occurrence time

Turning on Only warnings leaves only events with problems. It is usually best to keep it on. Most events are normal-operation events, and without it, problems are hard to find.

Repeated occurrences of the same event are grouped into one row, with only the Last seen time updated. A time that keeps changing means the problem is still ongoing.

Events disappear after about an hour by default. To investigate an older failure, look at the monitoring screen rather than events (see 9.2).

Namespaces​

View the list under Cluster > Namespaces.

Namespace list

A namespace is a unit for dividing resources. Dividing by team or environment prevents name collisions and lets you grant access permissions and resource limits separately.

ColumnDescription
NameThe namespace name
StatusActive means in use, Terminating means deletion is in progress
AgeTime elapsed since creation

Selecting the name shows the resources in that namespace, the resource limits applied (see 8.1), and the access permissions together.

Staying in Terminating for a long time means resources inside it are not being deleted. The cause is usually a resource connected to an external system, and it needs a check by operations staff.

Nodes​

View the list of servers making up the cluster under Cluster > Nodes.

Node list
ColumnDescription
NameThe node name
StatusWhether pods can be scheduled on it
Rolescontrol-plane (management) or worker
CPU · MemoryCurrent usage
VersionThe Kubernetes version installed on the node

Node Detail​

Selecting a node name opens the detail screen.

Node detail
SectionWhat it tells you
ConditionsThe items that determine whether the node is healthy
Capacity and allocatableThe resources the node has and the resources it can give to pods
System informationOperating system, kernel, container runtime versions
LabelsThe name tags on the node. Used in pod scheduling conditions
TaintsMarkers that make the node accept only certain pods
Scheduled podsThe list of pods running on this node

Look at these conditions.

ConditionHealthy valueIf it is otherwise
ReadyTrueThe node cannot accept pods
MemoryPressureFalseMemory is short
DiskPressureFalseDisk is short
PIDPressureFalseThe process count is at its limit

Capacity and allocatable are different. Capacity is everything the server has; allocatable is what remains after the share the system uses is subtracted. Pods are scheduled only within the allocatable range.

Not just any pod is scheduled on a node with a taint. If a pod is Pending while the node still has resources left, check the taints. Management nodes (control-plane) carry a taint by default, so general applications are not scheduled on them.

The Scheduling Disabled (Cordon) Marker​

If a Scheduling Disabled badge appears beside a node name, no new pods are scheduled on that node. It has been blocked ahead of maintenance.

Pods already running continue to operate. Only scheduling is blocked.

The Console does not display "drain complete". Kubernetes has no value meaning that state, so there is no way to distinguish a node that has finished draining from one where scheduling is merely blocked. Check whether the pods are actually gone in the scheduled pod list on the node detail.

Immutable OS Nodes​

From 1.0.4 the node operating system can be installed as an Immutable OS. The operating system is built and shipped as one piece, like a container image (image mode), and once deployed it cannot be modified.

What Gets Better​

BenefitWhy
Nodes become identicalThe operating system does not change, so node states cannot diverge. Fewer problems that happen "only on this node," and the same environment wherever a pod lands
Rollback happens in one stepThe operating system is one piece, so it is rolled back whole. No need to revert packages one at a time
Nothing drifts over timeNothing can be modified, so nothing accumulates. Package-installed nodes differ from each other after a few months

Two Things That Change in Return​

These change how work is done, so know them first. Both are the other face of "it does not change."

  • Additional packages cannot be installed on a node. dnf install does not work and system files cannot be edited. The same holds when you enter the node through the Console's web terminal. This is not a policy: the file system is read-only, so it is physically impossible — not even root can write.
  • Anything that used to be installed on the node runs as a container. Agents, monitoring tools, and utilities that previously sat on the node are deployed as DaemonSets or pods. This is also how nodes stay identical — nothing touches the node, so node states cannot diverge.

What You See on the Screen​

  • The System info — operating system field on the node detail shows a different value than on a package-installed node.
  • Updates involve a reboot. During it the node reads NotReady and pods move to other nodes. During a planned update this is normal.

Updating and rolling back are done by the operator. For the procedure, see the node operating system chapter of the installation guide.

What Immutable OS Gives You for Security​

Kept separate because it is used directly to meet security requirements.

Security benefitWhy Immutable OS makes it possible
SELinux is EnforcingThe image is fixed in that state, so it cannot be switched off while running. Package-installed nodes are disabled
Security settings survive a rollbackThe settings live in the image, not in /etc. Rolling back returns /etc to its earlier values, while settings inside the image remain
Vulnerability checklist items are already appliedBlocked root login, password policy, account lockout, restricted su, a 10-minute idle session limit, restricted source addresses, and a login banner ship inside the image. Operators configure nothing, and no node can be missed
SSH with a password is not possibleAccess is by key only. For the same reason, it applies identically on every node
The attack surface does not growArbitrary software cannot land on a node. What is installed is what the image contains

SELinux depends on the installation method. Within the same product, Immutable OS nodes are Enforcing while package-installed nodes are disabled. This difference applies only to nodes installed as Immutable OS.

When a Node Has a Problem​

When a node turns NotReady, its pods are moved to other nodes after a short while.

Order of checksWhere to look
1Conditions on the node detail — which one is False
2Events on the node detail
3Whether there is room on other nodes to move to (Allocated on the overview screen)

Fixing the node itself requires server access, so operations staff perform it. Pass on the node name and the condition status.