Course format

Observability 101 is a hands-on course. You learn by instrumenting a running microservice marketplace with your group, then by using what you built to investigate incidents on it.

1. 1Setup
   blind stack
2. 2Logs
   Logs
3. 3Metrics
   LogsMetrics
4. 4Traces
   LogsMetricsTraces
5. 5Connect
   LogsMetricsTracesSLOs
6. 6Incident
   LogsMetricsTracesSLOs

Your stack gains one layer of observability per session, then you use all of it.

How it works

• You work in groups of 4. Each group has its own deployed copy of the stack and its own Datadog organization.
• The stack runs and serves traffic from day one, but you can barely see inside it. Each session adds a layer of observability.
• Every session has lectures and a lab on your stack. How much time goes to each depends on the topic and on how the class is doing.
• Incidents happen on your stack during the course. Finding out what broke and why is part of the work.
• What you start in a session, you finish with your group before the next one. Each session builds on the previous one.

Lab sessions

Every session has a lab: the hands-on part, done with your group on your own stack. The session page lists the parts of the lab and what you hand in by the next session.

After a guided start in session 1, most of each lab is an investigation. You start from a symptom, such as a number in Datadog that does not add up or a ticket from a customer, and you find out what causes it. Look in Datadog first, and open the code only once the data points you somewhere. Every fix goes through a merge request on your team’s repository.

You write each investigation up as a Datadog Notebook, following the structure in Incident write-ups. In the first sessions the Notebook is short: what looked wrong, what you checked, what you changed, and a query showing the fix worked.

Later on, the Notebooks get closer to a post-mortem. The incidents are less obvious, you get less help finding where to look, and the write-up has to make sense to someone who was not there. The same six sections then also say when the problem started, what it broke for customers, how you found it, and what would catch it sooner next time: see Later write-ups. The session 6 exercise is written this way, and it is half of your grade.

Grading

Component	Weight	Scope
Incident investigations (sessions 2-5)	30%	Group
Instrumentation quality (sessions 2-5)	20%	Group
Final incident response (session 6)	50%	Group