Session 1: Why observability?

You have just inherited a marketplace that is already in production, with real customers placing orders every minute. The previous team left little documentation and almost no instrumentation. Over the next six sessions you will make this system observable, and use what you build to investigate incidents as they happen.

Today you get your bearings: what runs, where its data lands in Datadog, and how far you can trust what Datadog tells you.

SlidesThe lecture and the lab kickoff. Arrow keys to navigate, F for fullscreen.

Slides (PDF)The same deck as a PDF, one slide per page.

Before you come

Work through Before session 1 at home. This session assumes your accounts work and your tools are installed.

No local stack today

Everything in this session happens on your team’s deployed stack and in Datadog. Do not start the local Docker stack from the repository’s README.md: you do not need it today, and there is no time for it.

What you hand in

By the start of session 2:

1. A merged merge request on your team’s repository (see Part 4).
2. A Datadog Notebook named Session 1: inventory in your team’s organization (see Part 4).

Plan for the lab

1. Part 1Everyone is in
2. Part 2Meet your marketplace
3. Part 3A guided tour of Datadog
4. Part 4Challenge: the inventory
5. Part 5Bonus: logs and a ticketif you have time

Rotate who drives. By the end of the session, every member of the team should have run a Metrics Explorer query themselves.

Concepts you will use

Tags. A tag is a key:value pair, such as service:catalog, attached to every log line and metric point. You filter and group by tags, so most queries you write start with one.

Facets. In the Log Explorer, the filters on the left are called facets. Think of a facet as a column and of tag values as what is in that column: you filter on it, group by it and count it.

Unified service tagging. Datadog links logs, metrics and traces through three reserved tags:

Tag	Answers	In your stack
service	Which program produced this?	One per component, for example catalog
env	Which environment?	course for your deployed stack
version	Which build?	The Git commit your pipeline deployed

Datadog shows data under whatever tags it arrives with; it never checks them. On Kubernetes, the Datadog Agent reads these three tags from the tags.datadoghq.com/service, tags.datadoghq.com/env and tags.datadoghq.com/version labels of each pod. Your Helm charts set those labels.

How data reaches your organization. One Datadog Agent runs on your team’s Kubernetes node. It already collects every container’s standard output as logs, and every container’s CPU, memory and network as metrics, and ships them to your team’s organization. No application code is instrumented yet. That part is your job over the next sessions.

Your pod, for example catalog

Labels set by your Helm charttags.datadoghq.com/service: catalogenv: courseversion: <commit>

What the container emitsstandard outputCPU, memory, network usage

Datadog Agent, one per node

• reads the three labels as service, env, version
• adds tags from Kubernetes: kube_namespace, kube_deployment, pod_name, container_name, host

Your team's Datadog orgLogsMetricsTraces from session 4

What already works on day one: no application code involved, only labels and the Agent.

Infrastructure tags. Besides the three tags above, the Agent adds tags it discovers itself: kube_namespace, kube_deployment, pod_name, container_name, host and more. These come from Kubernetes, not from your charts.

INFOcatalog service listening

{"level":"info","msg":"catalog service listening","addr":":8082"}

• service:catalog
• env:course
• version:3f9c2e1…
• kube_namespace:t05
• kube_deployment:catalog
• pod_name:catalog-7d9f6c5b8-x2k4q
• container_name:catalog
• host:ip-10-0-12-34…

• Your Helm chart (pod labels)
• Kubernetes
• The node the Agent runs on

One log line from catalog as Datadog stores it. Values are examples.

Part 1: Everyone is in

Each member of the team, on their own laptop:

1. Run the setup check. Fix any FAIL line now, or tell the instructor.

2. Sign in to app.datadoghq.eu and check the organization name in the account menu is your team’s, on the EU1 site.

   

3. Open your team’s project on GitLab and find the latest pipeline on main.

4. If you have your kubeconfig:

   kubectl -n <team> get deployments
   kubectl -n <team> get pods -o wide

Part 2: Meet your marketplace

1. Open your storefront (your team hostname, from the welcome email), sign in (or create an account), and buy something.
2. Your order starts as pending. Refresh the order page until it changes.
3. Using the Architecture section of the repository’s README.md, write down which components your order went through, and in which order. Some of the work happened after the page answered you. Which part?
4. Make a list of every component your team runs: the services, the workers, the proxy, the datastores. Scroll through README.md, the directories under k8s/ and, if you have it, kubectl get deployments. Keep this list; you need it in Part 4.

Part 3: A guided tour of Datadog

Each step ends with questions. Write the answers down: they go in your Notebook later.

3.1 Infrastructure

1. Infrastructure, Host Map. How many hosts does your organization see? Your stack runs on one dedicated node. Does that match?
2. Infrastructure, Containers. Group by kube_deployment.
   • Which container uses the most memory right now? The most CPU?
   • Open one container and read its tags. Find service, env and version, and at least three infrastructure tags.

Note

The Kubernetes explorer is turned off in your organization. That is expected; everything you need today is in the views above.

3.2 Metrics

Open Metrics, Explorer. A query is one row with three fields, read left to right: the metric, then from (which data to keep), then how to combine it, such as avg by or sum by followed by a tag (one line per value of that tag). Always put your team’s namespace in from, for example kube_namespace:t05: (everywhere) also shows the cluster’s own components.

1. container.cpu.usage from kube_namespace:<team> avg by kube_deployment. Which deployment uses the most CPU? Then the same with container.memory.usage.
2. kubernetes.containers.restarts from kube_namespace:<team> sum by kube_deployment. Has anything restarted recently?
3. Open Metrics, Summary, the list of every metric your organization receives, and search container.. How many metrics does the Agent collect about containers? Pick one you did not know and find out what it measures.
4. Go back to the query from step 1 and change avg by from kube_deployment to service. Do both give you the same list of lines?

Grouped by kube_deployment, Postgres and Redis show up as N/A: they run as StatefulSets, not Deployments. That is expected.

3.3 Where the tags come from

1. In your repository, open k8s/catalog/templates/_helpers.tpl and find the tags.datadoghq.com/* labels. Then find where deployment.yaml uses them.
2. What will happen to every service’s version the next time your team merges a merge request? Check your answer after Part 4.

Part 4: Challenge: the inventory

Before you instrument anything, you need to know whether you can trust what Datadog already shows you. You write this challenge up as a Notebook, the way you will write up every incident in this course. Read Incident write-ups before you start.

Start the Notebook. In Datadog, open Notebooks, New Notebook and name it Session 1: inventory. Everything you find in this part goes into it.

Your task. Every component on your list from Part 2 should appear in Datadog under its own service. Check it with queries, not by reading long lists by eye:

1. In the Metrics Explorer, build two queries on the same metric. Each line of the result is one group, and its value is how many containers run in it:

   • A: kubernetes.containers.running from kube_namespace:<team> sum by kube_deployment
   • B: kubernetes.containers.running from kube_namespace:<team> sum by service

2. Compare A, B and your Part 2 list. Does every deployment have a service of the same name? Does a service count more containers than any single deployment?

3. When something does not add up, build a query whose result is the problem: for example a value of 0 where Kubernetes says 1. A query with no data shows nothing at all; the default zero function (in the query’s function menu) turns that into a 0 you can point at. Save this query to your Notebook as Evidence.

   

4. Find out why, and fix it the way your team will fix everything from now on. Then show the same query after the deploy as Verification.

Rules

• Every change goes through a merge request on your team’s repository. Nobody can push to main, and a merge request only merges when its pipeline passes. A merge to main deploys your stack.
• Investigate with Datadog first, and only then go to the code: the point is to learn what the tooling shows you, not to read the repository line by line.
• Data Datadog has already ingested keeps the tags it arrived with. Only data sent after your deployment shows the change.

You

1. BranchCommit your change on a branch.
2. Merge requestOpen it against main. Nobody can push to main.

Pipeline, before merge

1. Checksgofmt, go vet, golangci-lint, build, Go tests, helm lint.A failed check blocks the merge.
2. MergePossible only once the pipeline has passed.

Pipeline, after merge

1. Build imagesOne image per service, tagged with the commit.
2. Deployhelm upgrade --atomic, chart by chart, into your namespace.A failed deploy rolls back.
3. RunningNew pods replace the old ones.

The only way a change reaches your deployed stack.

Freeze the moment of the fix. Give your Evidence and Verification cells a fixed time frame, from a little before your merge to a little after the deploy, as described in Fixing a cell’s time frame. The pipeline in GitLab shows when the deploy ran: open it from the merge request or from Build, Pipelines.

Your Notebook follows the structure every write-up in this course uses:

Investigate

1. 1SymptomWhat looked wrong, in one or two sentences.
2. 2HypothesisWhat you think causes it, and how you would confirm it.
3. 3EvidenceA log query, a metric graph: what confirms or refutes it.live cell

Fix

1. 4ActionWhat you changed.MR link

Prove

1. 5VerificationThe system after your deploy.live cell
2. 6ConclusionThe cause in one sentence, naming the component and the setting.

Evidence disagrees with the hypothesis? Go back to step 2 and write a new one.

The structure of every incident Notebook in this course, including the final exercise.

Part 5: Bonus

Finished Part 4 early? Start on these. They lead into session 2.

5.1 A tour of the logs

Open Logs, Explorer, last 15 minutes.

1. On the left of the page is a list of filters, one per tag: Service, Host, Status and more. Datadog calls them facets. Open Service: it lists every service that sent logs, with a count. Which service writes the most log lines?
2. Under Visualize as, switch from List to Timeseries, a graph of how many lines arrive over time. Group it by service to get one line per service. Does the log volume of each service look steady, or does something burst?
3. Pick one line and open it. Find each tag from the diagram in Concepts and check its value. Which of them would change if the pod restarted? Which would change if your team deployed a new commit?
4. Reload a product page in your browser, then find the log line that page load produced. Search for its path as free text. Now try to find a log line for the order you placed in Part 2. What does it tell you that only some events are logged?
5. Open Live Tail, which shows log lines as they arrive, for a minute. Roughly how many lines per second reach your organization, and from which services?
6. Compare a line from web with a line from catalog. Which one could you filter on by field, and which one is just text? You will fix that in session 2.

5.2 A support ticket

A ticket from the merchandising team, who own the product catalog:

A customer says the Amazon Echo Show 10 has looked broken in the shop for a while. Everything else is fine. The catalog is ours, so don’t change anything: tell us what is wrong and we will fix it.

Write your diagnosis in a second Notebook, Session 1: bonus. Fill in Symptom, Hypothesis, Evidence and Conclusion. There is nothing to fix, so this one has no Action and no Verification. The Conclusion also explains why nothing in your stack reports an error.

Before session 2

• Your inventory merge request is merged, and its pipeline deployed.
• Session 1: inventory holds every section with live evidence.
• Every member can sign in to Datadog and GitLab, and has run a Metrics Explorer query.
• Session 2 is about logs. Look at one raw log line from three different services and decide, for each, how you would search for every request that failed.