Observability Three Signals: Logs, Metrics, Traces

Three signals

How they combine

Metrics say "something is wrong", traces say "which part", logs say "which line exactly".

Two pitfalls

• Logs but no metrics: when it breaks you scroll logs by hand, no alerting;
• Logs without a correlation ID: a request spanning services can't be tied together without a traceId.

Real-world cases: three times "we cannot find it"

1. Logs but no metrics: the outage ran for two hours before users reported it. With error-rate and P99 metrics, an alert would have fired early.
2. Logs without a correlation ID: a checkout spans five services, and without a traceId you can only stitch timestamps together. Instrument tracing and put traceId in logs, and one query reconstructs the whole path.
3. Unbounded metric dimensions: using userId or orderId as labels explodes the number of time series and crushes storage. High-cardinality fields belong in logs or span attributes.

FAQ

How much should I log? Decision-useful information: input summaries, key branches, durations and outcomes — not whole payloads or per-loop entries. Which metrics matter most? The golden signals: latency, traffic, errors, saturation. Will sampling miss incidents? It can; sample errors and slow requests at 100% and normal traffic proportionally. How do I keep alerts quiet? Alert on symptoms (error rate, latency) rather than single-host resource metrics, with severity tiers and suppression windows.

Roll it out in stages

The surest way to fail at observability is buying a platform before deciding what it is for. A sensible order:

1. Request IDs first: generate a traceId at the edge and propagate it everywhere — highest payoff, lowest cost;
2. Then the four golden signals: latency (P50/P95/P99), traffic, error rate and saturation, so incidents can at least be detected;
3. Then tracing: sample only critical entry points to keep storage and overhead sane;
4. Then log governance: agree levels, fields and retention, and put variables in structured fields instead of the message text.

Alert hygiene: keep the noise down

• Alert on symptoms, not causes: "checkout error rate > 2%", not "one host CPU > 80%";
• Severity and suppression: P1 pages, P2 notifies a channel, P3 goes on a dashboard only; suppress derivative alerts from the same incident;
• Write the first response steps: include dashboard links and diagnostic commands so the on-call does not hunt for docs;
• Review alert volume regularly: alerts nobody acts on should be downgraded or deleted, or the team learns to ignore all of them.

Cost and retention

Metrics grow linearly with retention and logs cost even more. A common split is two-tier metrics (high resolution short term, downsampled long term), 7–30 days of logs, and full trace detail only for slow and failed requests.

SLIs, SLOs and error budgets

• SLIs must be measurable: pick "share of successful requests" or "P99 latency" that comes straight from metrics, not "users are happy";
• Leave headroom in the SLO: target 99.9%, not 100% — the remaining allowance is the error budget;
• Let the budget drive decisions: plenty left, ship faster; exhausted, freeze features and fix reliability first;
• Use different windows: short (one hour) for alerting, long (30 days) for trend assessment.

Closing the loop after an alert

A full cycle is: alert fires, localise with traces and logs, mitigate, record the timeline, then add monitoring or tests. Skip the last step and the same failure returns the same way; a review should produce actionable changes, not a narrative of causes.

A shared convention for log levels

• ERROR: needs human action and should be alertable;
• WARN: anomalous but self-healing, useful for trends;
• INFO: key business events and state changes, kept in moderation;
• DEBUG: diagnostic detail, off by default and switchable on demand.

Once agreed, the level itself carries the response, so nobody has to read every line.