Incident write-ups

Every challenge and every incident in this course is written up as a Datadog Notebook in your team’s organization, from the session 1 inventory to the final exercise. The session page tells you the Notebook’s name, for example Session 1: inventory.

The structure

Every write-up has the same six sections, in this order:

Investigate

1. 1SymptomWhat looked wrong, in one or two sentences.
2. 2HypothesisWhat you think causes it, and how you would confirm it.
3. 3EvidenceA log query, a metric graph: what confirms or refutes it.live cell

Fix

1. 4ActionWhat you changed.MR link

Prove

1. 5VerificationThe system after your deploy.live cell
2. 6ConclusionThe cause in one sentence, naming the component and the setting.

Evidence disagrees with the hypothesis? Go back to step 2 and write a new one.

The structure of every incident Notebook in this course, including the final exercise.

• Evidence and Verification are live Datadog cells: real queries, never screenshots.
• Action links to the merge request that holds your fix. Every fix goes through a merge request on your team’s repository.
• If the evidence disagrees with your hypothesis, write a new hypothesis and keep the old one. A rejected hypothesis is part of the investigation.

Later write-ups

From the later sessions on, the write-up reads more like a post-mortem. The sections stay the same, but some of them say more:

• Symptom also says when the problem started and what it broke for customers.
• Evidence also shows how you found the problem, starting from the first signal you noticed.
• Conclusion also says what would have caught the problem sooner, such as a monitor, a log field or a metric you would add.

Adding a live cell

Build the query in the Log Explorer or the Metrics Explorer first. When it shows what you mean, use Export (or Save to Notebook) and pick your Notebook. The query lands in it as a cell you can still edit.

Fixing a cell’s time frame

By default a cell shows the Notebook’s time range, such as the past hour. That window moves with the clock, so the moment your fix deployed soon scrolls out of view.

For Evidence and Verification, set the cell’s own time frame to a fixed range: from a little before the problem or your merge to a little after your deploy. The pipeline in GitLab shows when the deploy ran. In the cell’s editor, under Set time preferences, pick Custom time; the pause button stops a relative range such as Past 1 Hour from moving. The cell stays a real query; it keeps showing that window whenever someone opens the Notebook.

What makes a good write-up

• Someone outside your team can follow it without asking you anything.
• Each claim points at a cell that shows it.
• The Conclusion names the component and the setting that caused the problem, in one sentence.