Quick Guide to VCF Automation for VCD Administrators

 

Kubernetes POD Troubleshooting Tactics

There’s a joke in the industry:

Debugged failed pods for 8 hours - No luck.

A random restart the next morning - all set!

If you’ve been there, you know the frustration.

But instead of hoping for a miraculous restart, here’s a structured way to troubleshoot Kubernetes pods effectively.

1. Check Logs

kubectl logs <pod_name>

If your pod has multiple containers, specify one:

kubectl logs <pod_name> -c <container_name>

2. Analyze Pod Status

kubectl get pod <pod_name>

Look at the STATUS column.

If it shows CrashLoopBackOff, ImagePullBackOff, or ErrImagePull, you have clear hints on what to check next.

3. Describe Pod

kubectl describe pod <pod_name>

Look for warning events, scheduling failures, and container state details.

4. Verify Pod Configuration

A misconfigured pod can cause all sorts of issues. Review its YAML configuration.

kubectl get pod <pod_name> -o yaml

Check environment variables, resource limits, image versions, and volumes.

5. Check Events

Kubernetes events provide historical context on failures.

kubectl get events --sort-by=.metadata.creationTimestamp

Pay attention to events like FailedScheduling, ImagePullBackOff, or OOMKilled

6. Validate Container Images

Ensure your container images are correct and available:

Check if the image tag exists.

kubectl get pod <pod_name> -o jsonpath='{.spec.containers[*].image}'

Try pulling the image manually.

docker pull <image_name>

7. Restart Pod

Sometimes, instead of deleting the pod, restarting the deployment helps.

kubectl rollout restart deployment/<deployment_name>

8. Review Service Dependencies

Pods may fail if dependent services are unavailable. Check the relevant services.

kubectl get svc

Ensure services are resolving correctly.

nslookup <service_name>

9. Check Network Connectivity

If your pod can’t communicate with another service, test connectivity.

kubectl exec -it <pod_name> -- sh

ping <target_host>

curl <target_url>

10. Inspect Resource Usage

If your pod is OOMKilled or throttled, check resource usage.

kubectl top pod <pod_name>

Compare with defined limits.

Following this structured approach, you save time, avoid frustration, and debug with confidence!