# Chaos Engineering

Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.

## 1. The Concept

It is not about "breaking things" randomly. It is about **controlled experiments** to identify weaknesses before they cause outages.

*   **Analogy:** A vaccine. You inject a small amount of harm (virus) to train the immune system (infrastructure) to handle the real thing.

## 2. Principles of Chaos Engineering

1.  **Define Steady State:** Understand what "normal" behavior looks like (metrics like latency, error rate).
2.  **Hypothesize:** "If we disconnect Database Node B, the system should failover to Node C within 2 seconds."
3.  **Introduce Variables:** Simulate real-world events (server crash, network latency, disk failure).
4.  **Try to Disprove Hypothesis:** If the system fails or steady state is lost, you found a weakness.

## 3. The Origin: Chaos Monkey

Netflix popularized this practice when moving to AWS. They built **Chaos Monkey** to randomly terminate instances in production to ensure their engineers built resilient services that could survive server failures automatically.

## 4. Common Experiments

*   **Resource Exhaustion:** Eat up 100% CPU or RAM on a server.
*   **Network Latency:** Add 500ms delay to all packets between Service A and Service B.
*   **Packet Loss:** Drop 5% of network packets.
*   **Time Travel:** Change the system clock on servers.
*   **Kill Processes:** Terminate critical processes or containers.

## 5. Tools

*   **Chaos Monkey:** The original random instance terminator.
*   **Gremlin:** A "Chaos-as-a-Service" platform.
*   **LitmusChaos:** Kubernetes-native chaos engineering.
*   **Chaos Mesh:** Open-source cloud-native chaos engineering platform.

## 6. Safety First

*   **Blast Radius:** Start small (one container, one user group) and expand only if successful.
*   **Stop Button:** Have a big red button to stop the experiment immediately if metrics go haywire.

[[programming/distributed-systems]]
[[programming/microservices-architecture]]
[[programming/software-testing-basics]]