Skip to content
build-3000
devops

Observability Foundation

Logs, metrics and traces that answer questions at 3am - plus alerts tied to what users actually feel.

Before this: Server-Side Validation and Errors

Objective

Instrument the Ledger so any incident can be diagnosed from evidence: structured logs with correlation ids, per-endpoint health metrics, a stated SLO with an alert rule, and an incident drill proving the tools work under fire.

Vocabulary

The words to use when directing your agent.

structured logging
Log lines written as queryable fields (JSON), not freeform sentences.
correlation id
An identifier shared by every log line belonging to one request's journey.
metric
A numeric measurement over time - request counts, error rates, durations.
trace
The connected story of one request across handlers, jobs and services.
SLO
Service level objective - the measurable reliability promise, like success rate targets.
alert
A condition on metrics that summons a human before users give up.

Understand it

Three pillars, pragmatically

Logs: the forensic record - structured, correlated, secret-free (Level 2's rules stand). Metrics: cheap numbers aggregated per endpoint - count, error count, duration summary - exposed on a metrics endpoint or simple stats page. Traces: correlation ids propagated from HTTP handler into queue jobs and outbound calls connect the whole journey. You do not need purchased platforms to learn these; you need the habits.

The incident drill

Observability untested is decoration. Inject a realistic failure into your own app - a dependency timeout, a webhook rejection storm - then use ONLY your logs, metrics and alerts to detect, bound and diagnose it. Time the detection. Note what you could NOT answer; those gaps are next sprint's instrumentation backlog.

Core concepts

Logs answer questions, not decorate terminals

Write logs for the investigator you will become: who did what, to which resource, with what outcome - as fields. 'Payment webhook rejected: signature invalid, event=evt_123' answers a question; 'error occurred' starts a archaeology dig. Every line carries the correlation id tying it to its request-mates.

Alert on symptoms, diagnose with causes

Users feel failed payments and slow pages - alert THERE. CPU spikes and disk noise are causes worth investigating, not paging people about. An SLO makes this concrete: state the target ('at least 99 percent of requests succeed over a rolling window'), then alert when burn threatens it. Everything else goes to dashboards, not phones.

Examples

The 3am question

'Did payment webhooks fail overnight?' Structured logs answer in seconds: filter event-type equals webhook, outcome equals rejected, last twelve hours - count, event ids, reasons. Freeform log files answer the same question with grep and prayers.

Following one request across worlds

User submits an expense; handler logs start with correlation id c-881; enqueue job logs carry the same id; worker completion carries it too. One search reveals the entire biography - request to consequence - which is precisely what you want during an incident.

Common mistakes

Common mistake

Logging everything at maximum verbosity 'to be safe', drowning signal in noise.

Fix

Log decisions, transitions and failures with intent. Volume without purpose hides the smoking gun inside haystacks.

Common mistake

Alerting on internal causes - CPU, memory, restart counts - instead of user-facing symptoms.

Fix

Anchor alerts to SLOs users would feel breaching. Causes feed investigation after the page, not the page itself.

Working with your AI agent

Commission a logging schema BEFORE more code ships: field names, types, forbidden contents (payloads, secrets), correlation-id propagation points. Then have your agent wire every route and job through it uniformly. Ask for the metrics endpoint listing per-route counts/errors/durations - and a README section mapping each metric to the question it answers.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Watchtower

Build the Ledger's nervous system: structured correlated logging across requests AND background jobs, a metrics endpoint exposing per-route health, a written SLO with an alert rule, and a timed incident drill using nothing but your own instrumentation.

Requirements

  • All routes and workers emit structured JSON logs carrying a shared correlation id per request-job chain
  • Metrics endpoint exposes per-route request count, error count and duration summary - plus job outcomes
  • SLO statement written: the reliability promise for this app, with the metric that measures it
  • Alert rule defined against SLO burn: threshold, window, and what the on-call human checks first
  • Incident drill performed and documented: injected failure, detection time, diagnosis narrative, unanswered questions list
  • Grep-audit proves zero payload/secret contents in logs

Context for your agent

Instrument my Expense Ledger for production observability. First agree with me on the log schema: fields, levels, correlation-id propagation (request into queued jobs into outbound calls). Wire ALL routes and the job runner through uniform structured logging; add /metrics returning per-route counters, error counts, duration summaries and job outcomes as JSON. Help me write the SLO statement and one concrete alert rule. Then coach my incident drill: inject a failing dependency, let ME drive detection and diagnosis using only Watchtower outputs, record timings and gaps.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/10
Functional
Architecture
Security
Testing
Production

Verification prompt - paste into your AI agent

Stress-test my Watchtower instrumentation as the on-call engineer during a simulated night incident - judge only what the system tells you, not what you can read from my code. I will trigger a hidden failure; use ONLY logs and metrics to: (1) detect that something is wrong and note how long detection took, (2) bound the blast radius - which routes and jobs are affected, (3) identify the failing component and current status, (4) decide the first mitigation action. Then grade PASS or FAIL each DoD item and list every question you COULD NOT answer with my observability - that list is my backlog.

Run this against “Observability Foundation” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. State your SLO and name the exact metric watching it.

  2. Trace one request's correlation id across handler, job and response.

  3. During the drill, what could your tools NOT tell you - and what will you add first?

You can now

  • Instrument systems for evidence-based incident response
  • Write SLOs and alert rules anchored to user-felt symptoms
  • Practice incidents calmly before production runs them for real

Up next

CI/CD Pipelines