Ephemeral Containers: Advanced Debugging

Key Takeaways for AI & Readers

• Debugging Distroless Images: Ephemeral containers provide a solution for debugging minimalist "distroless" images that lack common troubleshooting tools like shell, ls, or curl. They let you attach a fully-equipped debugging container to a running Pod without restarting it.
• Temporary Injection: They allow you to inject a temporary container with debugging utilities into a running Pod without restarting or modifying the original application container. Ephemeral containers cannot be removed once added -- they exist until the Pod is deleted.
• kubectl debug Utility: The kubectl debug command facilitates the creation of ephemeral containers, pod copies, and node debugging sessions. It is the primary interface for all ephemeral container operations.
• Process Namespace Sharing: When targeting a specific container with --target, the ephemeral container shares the process namespace of the target container, allowing you to inspect running processes, read /proc entries, and attach profilers to the application.
• Beyond kubectl exec: Ephemeral containers overcome the limitations of kubectl exec by providing their own shell, filesystem, and tools. When the target container has no shell binary, exec fails, but an ephemeral container brings its own.

Modern production pods often run distroless images -- minimal images that contain only your application binary and its essential runtime dependencies. They have no shell (sh), no ls, no curl, no strace. This is excellent for security (smaller attack surface, fewer CVEs) but creates a significant challenge when you need to debug an issue in production.

Ephemeral containers solve this by letting you attach a temporary debugging container to a running Pod, bringing whatever tools you need without modifying the application image or restarting the Pod.

1. What Are Ephemeral Containers?

Ephemeral containers are a special container type in the Kubernetes Pod spec. Unlike regular containers and init containers, ephemeral containers:

• Cannot be defined in advance: They are not part of the Pod spec at creation time. They are added to a running Pod via the API.
• Cannot be restarted: If an ephemeral container exits, it stays in a terminated state. You cannot restart it.
• Cannot be removed: Once added, an ephemeral container persists in the Pod spec until the Pod is deleted.
• Have no resource guarantees: Ephemeral containers do not have ports, readiness probes, liveness probes, or resource limits by default.
• Share the Pod's network and volumes: Like all containers in a Pod, ephemeral containers share the same network namespace (localhost), IPC namespace, and mounted volumes.

Ephemeral containers became generally available in Kubernetes 1.25.

2. Injecting a Debugger

Instead of restarting the Pod or changing its image, you attach a new debugging container to it. The debugging container has its own filesystem with all the tools you need, but it shares the Pod's network and (optionally) the process namespace of the target container.

Running Pod

🚀

Main Application

distroless (No shell)

🛠️

Ephemeral Debugger

/bin/sh, curl, gdb

Ephemeral Containers allow you to inject a temporary container into an already running pod. Perfect for debugging production "Distroless" images.

3. Using kubectl debug

The kubectl debug command is the primary interface for working with ephemeral containers. It supports three modes: attaching an ephemeral container to a running Pod, creating a copy of a Pod with modifications, and creating a debugging Pod on a specific node.

Attaching to a Running Pod

# Attach an ephemeral container with debugging tools to a running pod
kubectl debug -it my-app-pod --image=busybox:1.36 --target=app

# --image: The debugging image (busybox, nicolaka/netshoot, alpine, etc.)
# --target: Share the process namespace of this container
# -it: Interactive terminal (stdin + tty)

When --target=app is specified, the ephemeral container shares the process namespace of the app container. This means you can:

• Run ps aux to see all processes in the application container.
• Read /proc/<pid>/environ to inspect environment variables.
• Read /proc/<pid>/fd/ to see open file descriptors and network connections.
• Use strace -p <pid> to trace system calls (if the security context allows).

Using Specialized Debugging Images

Different debugging scenarios call for different images:

# Network debugging (curl, dig, nslookup, tcpdump, netstat, iptables)
kubectl debug -it my-app-pod --image=nicolaka/netshoot --target=app

# Full Linux toolkit (strace, ltrace, gdb, perf)
kubectl debug -it my-app-pod --image=ubuntu:22.04 --target=app

# Minimal debugging (shell, basic utilities)
kubectl debug -it my-app-pod --image=busybox:1.36 --target=app

# Python-based debugging (for custom scripts)
kubectl debug -it my-app-pod --image=python:3.12-slim --target=app

4. Debugging Crash-Looping Containers

When a container is in CrashLoopBackOff, it restarts too quickly for kubectl exec to connect. Ephemeral containers solve this, but an even better approach is to create a Pod copy with a modified entrypoint.

# Copy the pod and override the command to prevent the crash
kubectl debug my-app-pod -it --copy-to=my-app-debug \
  --container=app \
  --image=my-app:v2.1 \
  -- sh

# This creates a new pod "my-app-debug" with:
# - The same spec as the original pod
# - The "app" container's command overridden with "sh"
# - An interactive terminal

Now you have a copy of the Pod that does not crash, and you can inspect the filesystem, check configuration files, test database connections, and identify the root cause.

# Copy with shareProcessNamespace enabled for deeper inspection
kubectl debug my-app-pod -it --copy-to=my-app-debug \
  --share-processes \
  --image=busybox:1.36

5. Debugging Distroless and Minimal Images

Distroless images from Google (gcr.io/distroless/*) contain no shell. The kubectl exec command fails immediately because there is no /bin/sh to execute. Ephemeral containers are the only way to get an interactive shell for these Pods.

# The target container is distroless — exec fails
kubectl exec -it my-app-pod -- /bin/sh
# error: Internal error occurred: error executing command in container:
#   exec failed: unable to start container process:
#   exec: "/bin/sh": stat /bin/sh: no such file or directory

# Use an ephemeral container instead
kubectl debug -it my-app-pod --image=busybox:1.36 --target=app

# Now inside the ephemeral container:
# - The ephemeral container has /bin/sh from busybox
# - Process namespace sharing lets you see the app's processes
# - You can inspect /proc/1/root to access the app container's filesystem

Accessing the target container's filesystem through /proc/1/root (where PID 1 is the application process in the target container) lets you read configuration files, logs, and application data even though the target container has no shell.

6. Node-Level Debugging

kubectl debug can also create a privileged Pod on a specific node for debugging node-level issues like filesystem problems, kernel parameters, or networking configuration.

# Debug a specific node
kubectl debug node/worker-node-3 -it --image=ubuntu:22.04

# This creates a pod that:
# - Runs on the specified node
# - Has the node's root filesystem mounted at /host
# - Runs in the host PID and network namespaces

Inside the node debugging Pod:

# Access the node's filesystem
ls /host/var/log/
cat /host/etc/kubernetes/kubelet.conf

# Use chroot to fully enter the node's filesystem
chroot /host

# Now you can run node-native tools
systemctl status kubelet
journalctl -u kubelet --since "10 minutes ago"
crictl ps

7. Ephemeral Containers in the Pod Spec

When you add an ephemeral container via kubectl debug, Kubernetes updates the Pod's ephemeralContainers field in the spec. You can also create ephemeral containers programmatically by patching this field.

# Ephemeral container as it appears in the Pod spec after kubectl debug
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: app
      image: gcr.io/distroless/static:nonroot
      # ... original container spec
  ephemeralContainers:
    - name: debugger-abc12                   # Auto-generated name
      image: busybox:1.36
      command: ["sh"]
      stdin: true
      tty: true
      targetContainerName: app               # Process namespace sharing target
      securityContext:
        capabilities:
          add: ["SYS_PTRACE"]               # Needed for strace/gdb

# Programmatic ephemeral container creation via kubectl patch
# kubectl patch pod my-app-pod --subresource=ephemeralcontainers --type=strategic -p '
apiVersion: v1
kind: EphemeralContainers
metadata:
  name: my-app-pod
ephemeralContainers:
  - name: manual-debugger
    image: nicolaka/netshoot
    command: ["sh"]
    stdin: true
    tty: true
    targetContainerName: app
# '

8. Security Implications

Ephemeral containers introduce security considerations that platform engineers must understand:

RBAC Controls

The ability to create ephemeral containers is controlled by RBAC. Specifically, a user needs the patch verb on the pods/ephemeralcontainers subresource. You can restrict debugging to specific namespaces or roles.

# RBAC Role allowing ephemeral container creation
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-debugger
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["pods/ephemeralcontainers"]
    verbs: ["patch", "get"]                # Allow creating and viewing
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]                 # Allow listing pods to debug

Process Namespace Sharing

When an ephemeral container uses --target, it can see all processes in the target container, read their environment variables (which may contain secrets), and inspect their file descriptors. This is a powerful capability that should be restricted to authorized personnel.

Volume Access

Ephemeral containers share the Pod's volumes. If the Pod mounts a Secret or ConfigMap, the ephemeral container can read it. Be aware that debugging a Pod inherently grants access to its secrets.

Audit Logging

All ephemeral container creations are recorded in the Kubernetes audit log. Enable audit logging and alert on ephemeral container creation in production namespaces to maintain an audit trail of debugging activities.

9. Comparison: Ephemeral Containers vs. kubectl exec

Aspect	kubectl exec	Ephemeral Container
Requires shell in target	Yes	No (brings its own)
Works with distroless	No	Yes
Can add tools	No	Yes (custom image)
Process namespace sharing	No (same container only)	Yes (cross-container)
Persists after exit	No	Yes (until Pod deletion)
Works with CrashLoopBackOff	Rarely (timing-dependent)	Yes (via pod copy)
Node-level debugging	No	Yes
RBAC	pods/exec	pods/ephemeralcontainers

Use kubectl exec when the target container has a shell and the tools you need. Use ephemeral containers when it does not, or when you need tools that are not in the target image.

Common Pitfalls

1. Forgetting --target: Without --target, the ephemeral container does not share the process namespace. You can still use the Pod's network, but you cannot see the application's processes or access /proc entries.
2. Using overly large debug images: Pulling a 500 MB debug image on a node with limited bandwidth delays debugging. Keep a few small debug images (busybox, netshoot) pre-pulled on nodes or use imagePullPolicy: IfPresent.
3. Leaving ephemeral containers running: Ephemeral containers consume resources (memory, CPU) even after you detach. Since they cannot be removed, the Pod carries this overhead until it is deleted or rescheduled.
4. Security context restrictions: If the Pod's security context or PodSecurityAdmission policy is restrictive (no SYS_PTRACE capability, restricted profile), some debugging tools (strace, gdb) will not work inside the ephemeral container.
5. Confusing pod copy with ephemeral containers: kubectl debug --copy-to creates a new Pod; it does not modify the original. Changes and fixes in the copy do not affect the original running Pod.
6. Not enabling audit logging: In security-sensitive environments, ephemeral container creation should be audited. Without audit logging, debugging sessions are invisible to the security team.

What's Next?

• Practice debugging distroless containers with kubectl debug in a test environment before you need it in a production incident.
• Create a standard debugging image with your organization's preferred tools (curl, dig, strace, tcpdump, jq) and pre-pull it on cluster nodes.
• Configure RBAC to grant ephemeral container creation to on-call engineers and SREs while restricting it from general developers in production namespaces.
• Set up audit log alerts for ephemeral container creation in production to maintain security visibility.
• Explore node debugging with kubectl debug node/ for troubleshooting kubelet, container runtime, and networking issues at the node level.
• Review the EphemeralContainers feature gate documentation for any cluster-specific configuration requirements (the feature is GA since 1.25, but older clusters may need it enabled).