ChallengesreDestructive
Chaos: Failure Injection & Recovery — Challenge
Kill things deliberately, measure how long recovery takes, and find the assumption that was wrong.
- Time
- 25 min
- Level
- Advanced
- Objectives
- 4 objectives
- Cost
- Free
Before you start
You will need
- kind or minikube
- kubectl 1.28+
You will be able to
- Form a hypothesis before injecting a failure
- Measure recovery rather than observing it
- Recognise a self-healing gap that only appears under failure
You are done when
0 of 4
The goal#
Achieve the same outcome as Chaos: Failure Injection & Recovery, from an empty starting point, without the steps.
The architecture diagram says the system is highly available. Nobody has tested it.
Chaos engineering is not breaking things at random — it is stating what you believe will happen, then checking.
What must be true when you are done#
- A killed Pod is replaced automatically, and you measured how long it took.
- You found at least one case where recovery did not happen as expected.
- A PodDisruptionBudget prevents an eviction that would have caused an outage.
- Each experiment had a written hypothesis before it was run.
Rules#
- Do not open the guided lab until you are finished, or until the same problem has held you up for 20 minutes.
- Documentation is allowed and encouraged.
- Verify every criterion with a command whose output you can read.
If you get stuck#
- What did you expect, exactly?
- What happened instead — the error text, not a paraphrase?
- Which layer is that error from?
- What is the smallest command that proves the layer below is fine?
The concept behind it
Next up
Lab 55 of 58 on the project path