Waking Up at Midnight: The Danger of Underestimating Kubernetes Resource Limits
A few months ago, we migrated our service's backend from a Docker Compose environment to Kubernetes (EKS). The containerization process itself wasn't particularly difficult, and as deployment automation progressed, I felt a strong sense of confidence—thinking that operations would become much easier.
However, a few weeks after the migration, I was jolted awake in the middle of the night by a PagerDuty alert. The report indicated that pods were repeatedly entering a `CrashLoopBackOff` state.
When I checked the logs, the application appeared to be progressing through its startup sequence normally. There were no error messages. It was as if the process had been abruptly severed and vanished into thin air.
Ultimately, the culprit was our Kubernetes `Resource Limits` configuration.
The Trap of "Good Enough" Limits
In the Docker Compose era, because we were utilizing the full resources of the physical machine, we didn't have to worry much about memory or CPU constraints. However, in Kubernetes, the design of `requests` and `limits` is the cornerstone of stable operations.
In this case, I had set the memory `limits` slightly low under the assumption that it was "enough" for our needs. This proved fatal. Runtimes like Java and Node.js attempt to reserve a certain amount of memory during startup. When these requirements hit the Kubernetes `limits`, the OS (kernel) triggers the OOM Killer, forcibly terminating the process.
Because the termination happens at the OS level, the application is never given the chance to log an "Out of Memory" error. This makes debugging incredibly difficult, as there is no trail left behind by the software itself.
Below is a snippet of the manifest used when the issue occurred:
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-api
spec:
template:
spec:
containers:
- name: api-container
image: my-app:latest
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
# This was too low. This caused the OOMKilled error immediately after startup.
memory: "512Mi"
cpu: "500m"
If `requests` are set too low, you risk over-relying on available node resources, causing pods to become unstable the moment load increases. Conversely, if `limits` are set too low—as in this case—the process will die unexpectedly.
Lessons Learned from the Investigation
To resolve this issue, I first checked the status using `kubectl describe pod [pod-name]`. When I finally saw the text `Reason: OOMKilled`, it was a sobering moment.
From there, I began cross-referencing actual metrics (CloudWatch Container Insights) to map out the application's memory usage trends against the exact timestamps when the pods crashed.
Through this experience, I realized that three key points are critical when designing resources for containerized environments:
First, `requests` should be treated as a "guaranteed minimum." These should be set based on the actual baseline of the application during startup and steady-state operation. Conversely, `limits` must include a sufficient buffer to accommodate spikes in demand.
Second, you must account for runtime-specific behaviors. For example, when using the JVM, if the `-Xmx` setting conflicts with the Kubernetes `limits.memory`, the entire container will crash. It is vital to ensure consistency between the memory the runtime "thinks" it has and the actual limits imposed by the infrastructure.
Finally, monitoring is essential. Instead of just checking if a pod is "alive," you should graph the difference between `container_memory_working_set_bytes` and `container_spec_memory_limit_bytes`. Had I been monitoring this specific metric, I could have identified the warning signs before reaching the breaking point.
In the end, it took several days of high-pressure troubleshooting to revise the settings, increase the limits, and determine the appropriate thresholds through load testing. Container orchestration is incredibly powerful, but it demands a deep understanding of both the infrastructure and application layers to prevent these "silent deaths."