A system you can't see into is a system you can't run. When checkout gets slow at 2 a.m., someone needs to answer three questions fast: is it actually broken for users, where is it broken, and why. Observability is the practice of building systems so those questions have answers. You emit the right signals, keep them affordable, and alert people only when users are hurting.
\nIn interviews, monitoring usually shows up near the end (\"how would you know this is working?\") or as a dedicated question like \"design a metrics and alerting system\". A strong answer is specific. It names the few numbers that define health for this product, explains how alerts fire, and shows you know what it costs to collect everything.
","headings":[]},{"id":"metrics-logs-and-traces","title":"Metrics, logs and traces","html":"The three classic signals each answer a different question.
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Signal | What it is | Question it answers | Cost profile |
|---|---|---|---|
| Metrics | Numbers aggregated over time: counters, gauges, histograms, each with a few labels | Is something wrong? How much? Since when? | Cheap per event, since you store aggregates. Cost grows with the number of distinct label combinations. |
| Logs | One record per event, with whatever detail you choose | What exactly happened to this request or this user? | Expensive, since cost grows with traffic. |
| Traces | A tree of timed spans for one request as it crosses services | Where did the time go, and which hop failed? | Expensive per request, so usually sampled. |
They work as a funnel. A metric tells you the error rate on the payments service jumped at 02:10. A trace of a failing request shows the time going into a call to the fraud-check service. That service's logs, filtered by the trace id, show it timing out against a database replica. Connecting the three, mostly by putting the same trace id in logs and traces, is what turns them from three tools into one investigation.
\nContinuous profiling (sampled CPU and memory profiles from production) is increasingly treated as a fourth signal. It answers \"which code is burning the CPU?\", which none of the other three can.
\nYou can't graph everything usefully, so start from two checklists.
\nThe RED method (from Tom Wilkie) is for anything that serves requests. For every service, track:
\nThe USE method (from Brendan Gregg) is for resources: CPUs, disks, network links, connection pools, queues. For each one, track:
\nRED tells you users are affected. USE tells you which resource is the bottleneck. Google's SRE book uses a similar list called the four golden signals: latency, traffic, errors and saturation. For asynchronous pipelines (see message queues), add queue depth and especially the age of the oldest unprocessed message, which shows user-visible delay in a way depth alone doesn't.
\nThe gap between the SLO and 100% is the error budget. At 99.9% over 30 days:
\nThe budget turns reliability arguments into a number. If the budget is mostly unspent, ship faster and take risks. If it's gone, slow down and fix reliability. It also stops teams from chasing 100%, which no system reaches, and which costs a lot more than users notice. The availability numbers table in the primer shows how quickly each extra nine shrinks the allowed downtime.
\nAverages hide exactly the users you care about. Suppose 100 requests take these times: 95 take 50 ms and 5 take 2,000 ms.
\nSo report p50, p95 and p99 (and p99.9 for high-volume paths). Two practical rules follow.
\nFirst, you can't average percentiles. The p99 of a fleet isn't the mean of each host's p99. Record latency as a histogram with fixed buckets, which you can add up across hosts and then read percentiles from. Prometheus histograms work this way. Precomputed per-host quantiles (Prometheus \"summaries\") can't be combined correctly.
\nSecond, tails compound under fan-out. If a page makes 100 backend calls in parallel and waits for all of them, and each call is slow 1% of the time, the chance that at least one is slow is 1 − 0.99¹⁰⁰ ≈ 1 − 0.366 ≈ 63%. The backend's p99 becomes the page's typical case. That's why per-hop p99 matters even when it looks rare.
","headings":[{"id":"what-to-measure-red-use-and-slos","title":"What to measure: RED, USE and SLOs","depth":3},{"id":"red-and-use","title":"RED and USE","depth":4},{"id":"slis-slos-and-error-budgets","title":"SLIs, SLOs and error budgets","depth":4},{"id":"percentiles-not-averages","title":"Percentiles, not averages","depth":4}]},{"id":"alerting-on-symptoms","title":"Alerting on symptoms","html":"Every page should mean \"users are hurting, or soon will be, and a human must act now\". Anything else trains people to ignore pages.
\nWrite logs as key-value records, usually JSON, instead of free-form sentences. Then you can filter and aggregate them without fragile regexes.
\n{\"ts\":\"2026-03-04T02:10:41.512Z\",\"level\":\"error\",\"service\":\"payments\",\n \"msg\":\"fraud check timed out\",\"order_id\":\"o_88213\",\"timeout_ms\":800,\n \"trace_id\":\"7d3f9a12c4e85b60a1f2d3c4b5a69788\"}Always include the trace id. It's the join key between a log line and the request that produced it. Never log secrets, full card numbers or passwords, and treat personal data in logs as personal data, since logs get copied everywhere.
\nLogs and traces scale with traffic, so volume adds up quickly. Take a service at 20,000 requests per second that writes about 1 KB of logs per request:
\nTraces are worse. At 15 spans per request and roughly 500 bytes per span, 20,000 × 15 × 500 B = 150 MB/s, or about 13 TB per day. You sample:
\nA trace is one request's journey through the system. It's made of spans, one per unit of work (an HTTP handler, a database query, a queue publish). Each span records a start time, a duration, a trace id, its own span id and its parent's span id. The parent links let the trace view rebuild the tree and draw the familiar waterfall. Google's Dapper paper described this model, and most tracing systems since have followed it.
\nFor this to work across services, the context has to travel with the request. The W3C Trace Context standard defines a traceparent HTTP header:
traceparent: 00-7d3f9a12c4e85b60a1f2d3c4b5a69788-3e1f0a9b8c7d6e5f-01The four dash-separated fields are the version (00), the trace id (16 bytes as 32 hex characters, shared by every span in the trace), the parent span id (8 bytes as 16 hex characters) and flags (01 means sampled). Each service reads the incoming header, creates a child span and sends a new traceparent downstream with its own span id as the parent. A companion tracestate header carries vendor-specific data. For asynchronous hops, put the same context in message headers or metadata, so a trace can follow a request through a queue to the consumer that processes it.
OpenTelemetry is the vendor-neutral way to do all of this. It provides APIs and SDKs for most languages, automatic instrumentation for common frameworks, a wire protocol (OTLP) and the Collector, a service that receives, batches, samples and forwards telemetry to whatever backends you use. Instrumenting with OpenTelemetry means you can switch backends without touching application code.
","headings":[{"id":"logs-and-traces-in-practice","title":"Logs and traces in practice","depth":3},{"id":"structured-logs","title":"Structured logs","depth":4},{"id":"sampling","title":"Sampling","depth":4},{"id":"distributed-tracing","title":"Distributed tracing","depth":4}]},{"id":"cardinality-and-cost","title":"Cardinality and cost","html":"In a metrics system, every unique combination of metric name and label values is a separate time series, with its own storage and index entries. The number of series is the product of each label's distinct values. Take http_requests_total labelled by:
That's manageable. Now add a customer_id label with 10,000 customers. In the worst case that's 500,000 × 10,000 = 5 billion series. Even if only a fraction of those combinations ever occur, it's enough to make queries slow, run up a large bill or knock over the metrics backend. The Prometheus documentation warns against exactly this: don't put user ids, email addresses or other unbounded values in labels.
Rules of thumb:
\n/orders/{id}, never /orders/88213).You rarely have more than a few minutes for this, so be concrete:
\ntraceparent propagated through HTTP and queue messages, and trace ids in structured logs.If the question is \"design a metrics system\" itself, the deep dives are a write-heavy time-series store (append-only, compressed, partitioned by time), the collection model (agents pushing versus a server pulling, as Prometheus does), the rule evaluator that turns queries into alerts, and the cardinality limits that keep it alive.
","headings":[{"id":"monitoring-in-an-interview","title":"Monitoring in an interview","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"The mean hides the tail. A small fraction of requests could have become very slow while most got slightly faster, leaving the mean unchanged. Look at p95, p99 and p99.9 from a latency histogram, broken down by endpoint and region, then pull traces of slow requests to see which hop the time goes to.
\nCPU is a cause, not a symptom. High CPU might not hurt users at all, and users can be hurting while CPU is fine, so the alert will be noisy and miss real problems. Page on user-facing SLIs such as error rate and latency against an SLO, preferably on error-budget burn rate. Put database CPU on a dashboard for diagnosis, and maybe raise a ticket if it trends toward saturation.
\nThe budget is 0.05% of 200 million: 200,000,000 × 0.0005 = 100,000 failed requests. Spending all of it in 3 days instead of 30 is a burn rate of 10, so the error rate would be about 10 × 0.05% = 0.5%, assuming traffic is evenly spread.
\nEach distinct user id creates a new series for every histogram bucket and every other label combination, so the series count multiplies by the number of active users. That explodes memory and index size. Remove the label, and record the user id as an attribute on traces and in structured logs instead, where cost scales with events, not with distinct combinations. Then find a user's slow requests by searching logs or traces.
\nThe trace context isn't propagated across the asynchronous hop. The publisher must write the traceparent (and tracestate) into the message headers, and the worker must extract it and start its span as a child of, or linked to, the publishing span. OpenTelemetry instrumentation for many messaging clients does this, but custom queue code has to do it by hand.
traceparent and tracestate headers.