Skip to content
build-3000
backend

Background Jobs and Queues

Moving slow work out of the request path - workers, retries with backoff, and failures with a paper trail.

Before this: Servers and Backends

Objective

Move slow work out of HTTP requests into queued background jobs processed by workers, with retries that back off, failures that land somewhere visible, and job states users can see.

Vocabulary

The words to use when directing your agent.

job
A unit of work recorded to be done later, outside any waiting request.
queue
The ordered holding area where pending jobs wait.
worker
A separate process that picks up jobs and runs them.
backoff
Waiting progressively longer between retry attempts instead of hammering.
dead-letter queue
Where jobs go after exhausting retries - visible, not vanished.

Understand it

Anatomy of the pipeline

Request handler enqueues a job describing WORK TO DO (a payload), not doing it. A broker holds the queue - from a simple database table used as a queue to dedicated message brokers; start boring and local. A worker loop claims a job, runs it, records success, or records failure and reschedules with growing delays. After the final attempt, the job moves to dead-letter status where humans review it.

Visibility is half the system

Silent background work rots: nobody notices failures for weeks. Minimum viable observability: every job has a status (pending, running, done, failed, dead-letter), failed counts are glanceable, and each dead-letter job keeps its error and payload for diagnosis. Your L3 observability habits apply to queues from day one.

Core concepts

Short requests, deferred work

Anything slower than a breath - emails, report generation, file processing, calling slow third parties - belongs in a job. The request's new responsibility shrinks to: validate, enqueue, respond fast with a way to check progress later. Users get snappy responses; slow work gets room to fail and retry safely.

Jobs WILL run twice someday

Workers crash mid-job, restart, and pick the job up again. Most queue systems promise at-least-once delivery - meaning duplicate execution is a design assumption, not bad luck. Handlers that would corrupt state when rerun need guards now; the deep dive arrives in Level 4's idempotency module, but the awareness starts here.

Examples

Export without the spin

Generating a large CSV inline risks timeouts and ties up workers handling web requests. Instead: POST /exports creates an export job and returns 202 with a status URL. User polls or refreshes; when done, the download link appears. Slow became invisible.

Backoff in action

A flaky external dependency fails the job. Attempt 1 immediate, attempt 2 after ~30 seconds, attempt 3 after ~2 minutes, then dead-letter. The pattern survives transient blips without amplifying an outage - the difference between resilient and robotic.

Common mistakes

Common mistake

Sending email or generating reports inline in the request handler.

Fix

If any step could take seconds, enqueue it. Respond 202 + status endpoint. The request path owes users speed, not completion.

Common mistake

Jobs failing silently - no statuses, no dead letters, just vanishing work.

Fix

Job states persisted and viewable; exhausted jobs land somewhere a human reviews. Invisible failure is worse than failure.

Working with your AI agent

Require a job contract table before any queue code: job name, payload fields, triggering action, max attempts, backoff curve, dead-letter policy. One worker loop processes all jobs through one runner so logging, retries and statuses stay uniform - per-job bespoke retry logic is how queues become haunted graveyards.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Job Bench

Give the Ledger a real job system: CSV export and nightly summary generation move to background workers with retries, backoff, dead-letter review, and status endpoints - all observable, all restart-safe.

Requirements

  • POST creating an export enqueues a job and responds quickly with a status reference - no inline generation
  • Worker processes jobs with persisted status transitions: pending, running, done, failed, dead-letter
  • Failed jobs retry with growing backoff, then land in dead-letter retaining error details and payload
  • Status endpoint shows a job's current state; export completes with a downloadable artifact
  • Scheduled job produces the daily summary without any HTTP request triggering it
  • Queued jobs survive a worker restart - nothing lost, duplicates handled honestly

Context for your agent

Build a background-job system for my Expense Ledger using a boring local queue (database-table-backed is fine - propose options). Two producers: export endpoint (202 + status URL) and a scheduler tick producing the daily summary. One uniform worker loop with the job contract table we agree on first: payloads validated, bounded attempts, exponential-ish backoff, dead-letter retention with error text. Persist statuses; expose GET /jobs/:id. Simulate a failing job type to exercise retries and dead-lettering end to end.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/11
Functional
Architecture
Security
Testing
Production

Verification prompt - paste into your AI agent

Audit my Job Bench with suspicion about what happens when things go wrong - because in this system they are supposed to. Verify PASS or FAIL with evidence: (1) interrupt a running job (kill the worker) - does the job recover without duplicating side effects? Report exactly what you observed, (2) force repeated failure - does backoff follow the documented contract and does the job land in dead-letter with usable diagnostics, (3) can user A read user B's job status through any endpoint manipulation, (4) do queued jobs genuinely survive restart? Then tell me: which job, if silently lost right now, would upset users most - and how long until anyone noticed?

Run this against “Background Jobs and Queues” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. Which operation did YOU move out of the request path, and how did users' experience change?

  2. Why does at-least-once delivery mean your handlers must tolerate reruns?

  3. Where do your dead letters go, and who looks at them?

You can now

  • Move slow work into queues with honest status visibility
  • Configure retries and backoff that ride out transient failures
  • Keep background failures visible and diagnosable

Up next

Payments Integration