Challengelogging
Centralised Logging with Loki and Grafana — Challenge
Ship every Pod's logs somewhere they survive the Pod, then answer a real question with them.
- Time
- 25 min
- Level
- Intermediate
- Objectives
- 4 objectives
- Cost
- Free
Before you start
You will need
- kind or minikube
- Helm 3.14+
- kubectl
You will be able to
- Query logs by label instead of grepping a stream
- Correlate a metric spike with the log lines behind it
- Explain why Loki indexes labels rather than content
Cost — Free
on kind or minikube. On a cloud cluster Loki requests a persistent volume, billed per GB-month — see cleanup.
You are done when
0 of 4
The goal#
Achieve the same outcome as Centralised Logging with Loki and Grafana, from an empty starting point, without the steps.
A Pod crashed at 3am. It has been replaced, and kubectl logs shows the new one. The evidence went with the old container.
Metrics told you that something broke. Logs are how you find out why — but only if they left the node before the Pod did.
What must be true when you are done#
- Logs from every namespace are queryable in Grafana.
- You can filter to one Deployment's errors in the last 15 minutes.
- You produced a crash and found its cause from the logs alone.
- You can explain what happens to logs when a Pod is deleted, with and without Loki.
Rules#
- Do not open the guided lab until you are finished, or until the same problem has held you up for 20 minutes.
- Documentation is allowed and encouraged.
- Verify every criterion with a command whose output you can read.
If you get stuck#
- What did you expect, exactly?
- What happened instead — the error text, not a paraphrase?
- Which layer is that error from?
- What is the smallest command that proves the layer below is fine?
The concept behind it
Phase complete · 09 Observability
You can now: Metrics, logs, dashboards and alerts exist — and the alerts are ones a human can act on.
Next phase
Lab 50 of 58 on the project path