A Practical Guide to Docker and Kubernetes

Lessons from the Trenches: Mastering Production-Ready Operations and Optimization

In modern infrastructure, Docker and Kubernetes (K8s) are no longer just options; they have become the standard for building robust, scalable systems. However, when engineers move into the front lines of production, many encounter the same common hurdles:

"It works on my local machine, but it behaves strangely in production."
"Pods are stuck in a continuous loop of unexplained restarts (CrashLoop)."
"Image sizes have become so bloated that deployment times are unbearable."

These issues rarely stem from flaws in the technology itself. Instead, they usually arise from a lack of understanding of the underlying mechanisms hidden behind these layers of abstraction.

In this article, I will delve into the "gotchas" that aren't always covered in textbooks but frequently cause headaches in real-world operations. Rather than just listing commands, we will explore the internal structures to understand why these problems occur and how to solve them effectively.

---

1. Docker Pitfalls and the Art of Build Optimization

In the early stages of using Docker, it is easy to feel satisfied just by writing a `Dockerfile` that successfully completes a `docker build`. However, if you are designing for a production environment, the following points are critical elements that cannot be ignored.

The Trap of Layer Structure and Build Cache

Docker images are structured as a series of "layers" stacked on top of each other based on each instruction (RUN, COPY, ADD, etc.). A common mistake made by beginners is copying the entire directory—including frequently updated source code—and then running dependency installations in the same step.

# [Poor Practice]
FROM node:18
WORKDIR /app
COPY . .
RUN npm install

With this configuration, even a single character change in your source code will invalidate the cache for all subsequent layers (in this case, `npm install`), forcing the system to recalculate them every time. In a CI/CD pipeline, this leads to bloated build times—a major source of friction in professional environments.

The golden rule to avoid this is using "Multi-stage Builds" and "Strategic Copy Ordering."

Optimization via Multi-Stage Builds

In a production-ready setup, you should completely isolate the build environment from the runtime environment. This allows you to create lightweight, clean images that do not contain unnecessary build tools or cache files.

# --- Build Stage ---
FROM node:18-alpine AS builder
WORKDIR /app
# Copy only the dependency definition files first (maximizing cache efficiency)
COPY package*.json ./
RUN npm install
# Only then copy the rest of the source code
COPY . .
RUN npm run build

# --- Production Stage ---
FROM node:18-alpine
WORKDIR /app
# Extract only the necessary artifacts from the builder stage
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/main.js"]

The benefit of this approach isn't just a smaller image size. It also provides a significant security advantage by minimizing the attack surface and keeping the runtime environment clean. Additionally, properly configuring `.dockerignore` to ensure that `node_modules` or `.git` folders are not included in the build context is an essential practice for any professional workflow.

Handling Environment Variables (ENV vs. ARG)

Assuming that all environment variables can be handled via `ENV` is a dangerous design choice.

• ARG: Variables valid only during the build process. These are used for options during image creation (e.g., specifying versions).
• ENV: Variables persisted during container runtime. These are used to control the behavior of the application.

In production, you must never hardcode sensitive information like passwords or API keys in `ENV`. These should be injected from external secret managers, such as Kubernetes Secrets or AWS Secrets Manager.

---

2. Navigating Kubernetes "Debugging Hell" and Finding Solutions

While Docker is a technology for running individual containers, Kubernetes is a massive platform designed to orchestrate (manage and operate) those containers. However, because of its high level of abstraction, the difficulty of troubleshooting can spike significantly.

The Reality Behind "Why Won't My Pod Start?"

When you run `kubectl get pods` and see `CrashLoopBackOff` or `Pending`, it can be confusing. These are warnings from Kubernetes, but they hide specific underlying causes:

#### ① ImagePullBackOff / ErrImagePull
This is often a simple configuration error. However, it doesn't just mean "you typed the name wrong." It often involves missing authentication credentials (missing ImagePullSecrets) or improper tag usage (the reason why you should use specific version tags instead of `latest`).

#### ② OOMKilled (Out of Memory)
If a Pod restarts suddenly, the cause is often related to resource limits. In Kubernetes, if `resources.limits.memory` is not properly designed, it can overwhelm the node's memory, causing the Linux kernel to forcibly terminate the process (OOM).

#### ③ Confusing Liveness Probes and Readiness Probes
This is one of the most common pitfalls.

• Liveness Probe: Checks "Is the container alive?" If it fails, Kubernetes restarts the Pod.
• Readiness Probe: Checks "Is the application ready to receive traffic?" If it fails, traffic from the Service to that Pod is blocked.

For example, if a Liveness Probe fails while waiting for a database connection, the app will restart repeatedly before it can even start up (CrashLoop). Conversely, if a Readiness Probe stops responding during a heavy batch process, the load balancer will pull the pod out of rotation. Failing to design these correctly leads to an unstable system.

Practical Deployment Example

Below is a production-ready configuration that incorporates proper resource limits and probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
      - name: main
        image: my-repo/api-gateway:v1.2.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "512Mi"
            cpu: "1000m"
        readinessProbe:
          http:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          http:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

---

3. Comparison and Selection: Docker Compose vs. Kubernetes

The question "When should I actually use Kubernetes?" is a critical one that engineers face constantly. You do not need to run every environment on K8s.

Selection Criteria

• Development Environments: Docker Compose is overwhelmingly superior. It allows you to spin up containers simply and focus on local debugging.
• Small-scale Web Services: If your goal is to minimize operational overhead, there are many cases where Docker Compose or a single server execution is sufficient.
• High Availability & High Traffic: These are the points where you should choose Kubernetes. Features like auto-scaling, rolling updates, and self-healing have limits when managed manually.

---

4. Conclusion