When we built a management UI for Kubernetes, we didn’t include swap in the node monitoring view.
The reasoning was simple: swap was supposed to be disabled on Kubernetes nodes. If nobody was going to use it, there was no reason to make room for it in the dashboard.
Recently, I read an article on the Kubernetes blog, Scaling Kubernetes Workloads with Node Swap, and it brought that decision back to mind.
The article describes using local SSDs for swap so a single node can host more browser and Python sandboxes. These containers can hold a lot of memory while spending much of their time waiting for the user’s next instruction.
My first thought was: we probably need to add that missing piece to the dashboard after all.
Why we thought swap wasn’t needed
There was a good reason for that assumption. On Linux, the kubelet refuses to start by default when swap is enabled. Disabling swap has long been a familiar step when setting up a Kubernetes cluster.
And swap does affect performance. If a process needs data that has been swapped out to disk, it has to wait for that data to be read back into memory. Under memory pressure, that can happen repeatedly, which is especially troublesome for services that are sensitive to latency.
So leaving swap out of the UI seemed reasonable at the time. But over time, “disable swap on the nodes” easily became “Kubernetes doesn’t need swap.” When we thought about node monitoring later, we simply passed over it again.
Kubernetes has actually been working on node swap support for several years. It entered Alpha in v1.22 and reached GA in v1.34. Demand for AI sandboxes has brought more attention to the feature.
The containers are waiting, but the memory is still occupied
When we talk about AI infrastructure, GPUs tend to come to mind first. But an agent may also need to launch a browser, run Python, or execute external code in an isolated sandbox. Those environments bring their own CPU and memory overhead.
These containers often need a fair amount of memory while they’re busy. Once the task is finished, though, they can’t necessarily exit: they need to preserve the session and execution state for the next interaction.
One or two sessions aren’t much of a problem. With many sessions, the waiting containers can occupy a large share of the node’s memory. The node may run out of RAM before it runs out of CPU capacity.
But keeping data around doesn’t mean a process is constantly accessing it. While a session is waiting, some memory pages may go untouched for a long time.
That’s where swap can help. Linux can swap out pages that aren’t currently needed, freeing RAM for other Pods. When the original session accesses those pages again, they’re read back into memory. The application continues accessing memory as usual; the operating system handles the paging.
Of course, reading the data back takes time. Whether that delay is acceptable depends on the application.
More Pods, with some tradeoffs
The article tests compilation jobs, browsers, and Python sandboxes with swap backed by local SSDs. One result that stands out is the gVisor Python sandbox test: capacity increased from 80 sessions to 240, three times the original number.
Another result is worth reading alongside that one. The compilation workload performed well with its RAM limit reduced to 300 MB. Reducing it further to 200 MB increased execution time by more than 40%. The authors explain that, at that point, even the active working set was being forced into swap.
These are the authors’ benchmark results. I haven’t repeated the tests in my own environment, and I wouldn’t use that threefold increase directly when planning capacity for another application.
The underlying conditions have improved. cgroup v2 can account for and limit memory and swap separately, and local NVMe SSDs are much faster than spinning disks. But if frequently accessed data keeps getting swapped out and read back in, performance will still suffer.
So I’d start by understanding the workload: how many sessions are usually waiting, how much memory is temporarily unused, and what happens when those sessions all become busy at once. Much of the benefit in the article comes from data that can stay out of RAM for a while.
What I’d add to the monitoring view
Coming back to the dashboard we built, I’d start with swap capacity, usage, and trends. But I wouldn’t stop at a usage percentage.
Once workloads are allowed to use swap, seeing some usage isn’t necessarily a bad thing. We also need to know which Pods are using it, how often pages are being swapped, and whether the application is slowing down.
| Information | What it should help us understand |
|---|---|
| Node swap status, capacity, and usage | Is swap configured, and how much space is left? |
| Pod swap policy and actual usage | Which workloads can use swap, and which are using it? |
| Swap-in and swap-out rates | Are pages moving frequently between RAM and storage? |
| Memory pressure, storage latency, and application latency | Is paging affecting the node or the application? |
For example, a node may consistently have high swap usage, but most of that data is temporarily unused and there is little paging activity. Another node may have lower usage but keep swapping pages while application response times get worse. Looking only at percentages makes the second case easy to miss.
Kubernetes already provides swap metrics at the node, Pod, and container levels. The kubelet’s /metrics/resource endpoint can expose the following four metrics. All are measured in bytes and represent current usage or a limit:
| Metric | Level | Meaning |
|---|---|---|
node_swap_usage_bytes |
Node | Current swap usage of the node |
pod_swap_usage_bytes |
Pod | Current swap usage of the Pod |
container_swap_usage_bytes |
Container | Current swap usage of the container |
container_swap_limit_bytes |
Container | The container’s swap usage limit |
Pod metrics have namespace and pod labels; container metrics also have a container label. That lets us go from a node overview down to an individual container and see both its swap usage and its limit.
One detail to keep in mind when integrating them: NodeSwap has reached GA, but these four monitoring metrics are still marked ALPHA. They don’t yet have the compatibility guarantees of stable metrics. Before adding them to a dashboard, we need to check that the cluster and collection pipeline expose the data, and watch for changes when upgrading.
These four metrics cover usage and limits. Swap-in and swap-out rates need to come from other sources, such as a node exporter, and should be viewed alongside latency.
The UI should also distinguish between “swap isn’t configured,” “Pods aren’t allowed to use swap,” and “no data was collected.” Showing all three as zero might look tidy, but it won’t help much when troubleshooting.
A few things the UI should keep separate
Configuration matters here too. Setting failSwapOn: false only allows the kubelet to run on a node with swap enabled. Allowing eligible containers to use swap also requires LimitedSwap.
The scheduler doesn’t currently treat swap as additional schedulable memory, either. So the dashboard shouldn’t add RAM and swap together and label the result “available node memory.” They differ in both performance and scheduling behavior. The official swap management documentation covers the configuration in more detail.
Beyond the technical details, what stayed with me after reading the article was that missing part of our dashboard. We thought we’d never need it, so we didn’t give it much more thought.
Now there are new workloads and support for running them with swap, and the node monitoring view needs to catch up. If I were adding it today, the first things I’d want to see would be which Pod is using swap, and how much the application slows down when paging becomes frequent.
