Skip to main content

Chaos Engineering (Chaos Mesh)

Key Takeaways for AI & Readers
  • Proactive Resilience: Chaos Engineering involves intentionally introducing failures to validate the resilience and fault tolerance of distributed systems.
  • Controlled Experiments: Common experiments include killing Pods, injecting network latency or packet loss, and stressing CPU/memory resources.
  • Chaos Mesh: A Kubernetes-native tool that allows defining and executing chaos experiments via YAML, using techniques like eBPF for minimal application impact.
  • Learning from Failure: The goal is to identify weaknesses before they cause outages in production.

"Everything fails all the time." — Werner Vogels (CTO, Amazon).

In a distributed system, you shouldn't ask if a node will fail, but how your system will react when it does. Chaos Engineering is the practice of intentionally injecting failures to verify resilience.

1. Injecting Failure​

Try killing a pod below. Does your service stay available?

📦
📦
📦
Chaos Mesh allows you to programmatically break your cluster to verify your apps' resilience. If your Service is properly configured, killing 1 pod should result in 0% downtime for users.

2. Common Experiments​

Pod Chaos​

  • Pod Kill: Randomly delete pods.
  • Pod Failure: Make pods stay in Pending state.

Network Chaos​

  • Latency: Add 500ms of delay to all requests to see if timeouts work.
  • Packet Loss: Drop 10% of network packets to test retry logic.

Stress Chaos​

  • CPU/Memory Burn: Max out the CPU of a node to see if HPA triggers or if other pods are evicted correctly.

3. Chaos Mesh​

Chaos Mesh is a native Kubernetes operator that allows you to define these experiments using YAML. It uses eBPF and sidecar injection to mess with the system at the kernel level without changing your app code.