Skip to content

Observability and reliability

Observability and reliability: logs, metrics and traces, percentiles, OpenTelemetry, SLOs and error budgets, incident response and postmortems.

8 lessons

Signals

  1. 1Logs, metrics and tracesChoose the evidence that contains the fact, and count the series your metric attributes create.
  2. 2Structured logs and correlation IDsOne field name everywhere, the id read from the request's own store on every line, and secrets removed by the logger itself.
  3. 3Metric types and percentilesRecord durations in a histogram, add the instances' buckets, then read the percentile: a mean or an average of percentiles hides the slow requests.
  4. 4Distributed tracing and OpenTelemetryContext crosses a process only when the sender injects it and the receiver extracts it and starts its span there; queues too.

Reliability targets

  1. 5SLIs, SLOs, error budgets and burn-rate alertsCount good over valid events, target below 100%, and spend the gap as a budget: page on a fast burn, ticket a slow one.

Incidents

  1. 6Incident response and on-callOne commander coordinates; mitigate first on the fault's location alone; root-cause later; review blamelessly; keep on-call shares small.
  2. 7Blameless postmortemsTimeline on one clock, impact from the users' side, several causes, and owned actions that change the system.
  3. 8Debugging production systematicallySymptom metric first, then what changed, then RED for services and USE for resources; test one hypothesis against the data before touching anything.