k8s Troubleshooting

More:

• kubectl cheat-sheet
• k9s demo

Pod Troubleshooting

kubectl debug reference

Use kubectl debug to create an ephemeral container using, e.g. docker image aws-cli-v2-on-ubuntu:

> kubectl debug -it keel-7b5fbd4bcb-mlqbz --namespace=keel --target=mtail \
    --image=zulhfreelancer/aws-cli-v2-on-ubuntu
Targeting container "mtail". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-nwff9.
If you don't see a command prompt, try pressing enter.

root@keel-7b5fbd4bcb-mlqbz:/home# env|grep AWS

root@keel-7b5fbd4bcb-mlqbz:/home# aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

root@keel-7b5fbd4bcb-mlqbz:/home# aws sts get-caller-identity

[Errno 2] No such file or directory: 'less': 'less'

root@keel-7b5fbd4bcb-mlqbz:/home# export AWS_PAGER=""

root@keel-7b5fbd4bcb-mlqbz:/home# aws sts get-caller-identity
{
    "UserId": "12234454545454:i-121212121212",
    "Account": "84232323",
    "Arn": "arn:aws:sts::84232323:assumed-role/Karpenter-test-20231003172341681800000003/i-121212121212"
}

Alternatively use busybox image:

kubectl debug -it keel-7b5fbd4bcb-mlqbz --namespace=keel --target=mtail \
    --image=busybox:1.28

Example use of the ephemeral container with a psql client - note the way to specify a custom container entry point:

kubectl debug -it keel-58ffc64f8b-f9gk4 --namespace=keel \
    --image=jbergknoff/postgresql-client -- sh

Example use of the ephemeral container with a mongosh client:

kubectl debug -it foo-769f64b94-7frn4 \
    --image=alpine/mongosh --namespace=foo -- bash

Troubleshooting Network Connectivity Between Services

1. Check service discovery: Ensure you’re using the correct service name and namespace

kubectl get _service_ -n _namespace_

1. Test connectivity: Use an ephemeral debug container with networking tools

kubectl debug -it your-pod-name --image=nicolaka/netshoot --namespace=_namespace_

1. Check network policies: Network policies might be restricting traffic

   kubectl get networkpolicies -n _namespace_
   

2. Inspect ingress/HTTPRoute configurations: for external connectivity issues.