Troubleshooting#

Everything that follows builds on one rule: status codes first, logs second, describe always. After writing dozens of deployments in the v1 notes, here’s the operator-grade field guide.

Pod Status Quick Reference#

Exit Code Cheat Sheet#

The exit code in Last State: Terminated usually tells the whole story:

137 and 143 are just 128 + signal number. Seeing 137 on a memory-limited pod? That’s the kernel enforcing your limit - go re-read the scaling notes on limits vs requests.

The Golden Path#

# 1. Hangi pod, hangi durumda?
kubectl get pods

# 2. Detay + Events - okunacak en zengin kaynak
kubectl describe pod <pod-name>

# 3. Loglar; çöken pod'un önceki attempt'ı için --previous
kubectl logs <pod-name>
kubectl logs <pod-name> --previous
kubectl logs <pod-name> -c <container-name>

# 4. İçeri girip elle dene
kubectl exec -it <pod-name> -- sh

# 5. Cluster geneli olay akışı (neden Pending olduğunu burada da görürsün)
kubectl get events --sort-by='.lastTimestamp'
kubectl get events --field-selector reason=FailedScheduling

Useful structured queries once describe walls of text get tiring:

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" exit="}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl get events --field-selector involvedObject.name=<pod>

And when you’re unsure what a field even is:

kubectl explain pod.spec.securityContext --recursive | less

Don’t forget get endpoints <service> for “service returns 404/connection refused” mysteries: an empty endpoints list means the service’s selector matches no pods - the app may be perfectly healthy. Half of “broken ingress” bugs are selector mismatches, not app bugs.

kubectl debug: Debugging Without a Shell#

exec requires a shell in the image - distroless images have none, and adding tools to prod images is bad practice anyway. Ephemeral containers solve this:

kubectl debug -it <pod-name> --image=cilium/netshoot:latest --target=app

What happens: a temporary container joins the pod’s process namespace, so you can inspect the target container’s processes (ps aux) - no shell required in the target image. --target even shares the target’s process namespace specifically.

For node-level problems (disk, kubelet, network interfaces):

kubectl debug node/<node-name> -it --image=ubuntu:22.04

You get a container on that node with the host’s filesystem mounted at /host - the node’s logs live at /host/var/log, kubelet’s journal via journalctl (run chroot /host first for full tooling).

Türkçe homelab checklist (kafam karıştığında sırayla uyguladığım liste):

1. Pod durumu ne? (get pods)
2. Events ne diyor? (describe - Events ve Last State bölümleri)
3. Log’da stack trace var mı? (logs --previous)
4. Exit code ne? (yukarıdaki tablo - 137 = OOM, 127 = command yazım hatası)
5. Env var’lar configmap/secret’tan geliyorsa, o key gerçekten var mı?
6. Selector’lar eşleşiyor mu? (deployment selector ↔ pod labels; service selector ↔ endpoints)
7. PVC bağlıysa mount path ve izinler doğru mu? (fsGroup hatırlatması: security-context)
8. İzinler/root sorunları distroless image’larda mı çıktı? (distroless notları)

How to Diagnose a CrashLoopBackOff#

1. Cause one: point the api deployment at a ConfigMap key that doesn’t exist (or delete a key it needs), then watch:

kubectl get pods --watch

2. Read the state machine:

kubectl get pod <api-pod> -o jsonpath='{.status.containerStatuses[0].state}{"\n"}'

waiting.reason: CrashLoopBackOff - and restartCount climbing.

3. Get the last attempt’s story:

kubectl logs <api-pod> --previous | tail -n 20

4. Cross-check the exit code in describe:

kubectl describe pod <api-pod> | grep -A5 "Last State"

The distinction between logs and describe is where beginners lose time: logs shows what the app printed before dying; describe’s Events and Last State show what k8s observed. A container that dies before printing anything has an empty logs and a telling describe - missing env, missing file, bad command (exit 127).

5. Fix the root cause (restore the ConfigMap key), and watch restartCount reset with the new pod. Note: k8s does not reset the backoff timer for the same pod; after repeated fixes, delete pod is faster than waiting out a long backoff.

How to Debug a Shell-less Container#

1. Distroless-style: attach a toolbox container to a running pod and share process visibility:

kubectl debug -it <api-pod> --image=cilium/netshoot:latest --target=synergychat-api

2. Inside the debug container, inspect the target’s processes:

ps aux

You’ll see the target container’s process tree (thanks to --target + process namespace sharing).

3. Test network reachability from inside the pod’s network namespace - the debug container shares the pod’s IP, so this tests exactly what the app pod sees:

curl --max-time 3 http://api-service:8080/healthz

4. Cleanup happens automatically when you exit (ephemeral containers don’t survive pod restarts) - verify with kubectl get pod <pod> -o jsonpath='{.spec.ephemeralContainers}'.

How to Trace Scheduling Failures#

1. Reproduce a Pending from the v1 notes - set an absurd resource request on testram:

kubectl set resources deployment/synergychat-testram --limits=memory=40000Mi --requests=memory=40000Mi
kubectl get pods --watch

2. Filter the noise out of global events:

kubectl get events --field-selector reason=FailedScheduling

You’ll see exactly why: 0/1 nodes are available: 1 Insufficient memory.

3. Confirm the decision logic in describe (the Events section repeats the same verdict with more context), then reason about the fix path: lower the request, add a node (minikube node add), or remove a taint that blocks the only candidate node.

4. Clean up and verify recovery:

kubectl set resources deployment/synergychat-testram --limits=memory=256Mi --requests=memory=128Mi
kubectl get pods

for more control-plane-troubleshooting