Background Jobs and Queues
Moving slow work out of the request path - workers, retries with backoff, and failures with a paper trail.
Before this: Servers and Backends
Objective
Move slow work out of HTTP requests into queued background jobs processed by workers, with retries that back off, failures that land somewhere visible, and job states users can see.
Vocabulary
The words to use when directing your agent.
- job
- A unit of work recorded to be done later, outside any waiting request.
- queue
- The ordered holding area where pending jobs wait.
- worker
- A separate process that picks up jobs and runs them.
- backoff
- Waiting progressively longer between retry attempts instead of hammering.
- dead-letter queue
- Where jobs go after exhausting retries - visible, not vanished.
Understand it
Anatomy of the pipeline
Request handler enqueues a job describing WORK TO DO (a payload), not doing it. A broker holds the queue - from a simple database table used as a queue to dedicated message brokers; start boring and local. A worker loop claims a job, runs it, records success, or records failure and reschedules with growing delays. After the final attempt, the job moves to dead-letter status where humans review it.
Visibility is half the system
Silent background work rots: nobody notices failures for weeks. Minimum viable observability: every job has a status (pending, running, done, failed, dead-letter), failed counts are glanceable, and each dead-letter job keeps its error and payload for diagnosis. Your L3 observability habits apply to queues from day one.
Core concepts
Short requests, deferred work
Anything slower than a breath - emails, report generation, file processing, calling slow third parties - belongs in a job. The request's new responsibility shrinks to: validate, enqueue, respond fast with a way to check progress later. Users get snappy responses; slow work gets room to fail and retry safely.
Jobs WILL run twice someday
Workers crash mid-job, restart, and pick the job up again. Most queue systems promise at-least-once delivery - meaning duplicate execution is a design assumption, not bad luck. Handlers that would corrupt state when rerun need guards now; the deep dive arrives in Level 4's idempotency module, but the awareness starts here.
Examples
Export without the spin
Generating a large CSV inline risks timeouts and ties up workers handling web requests. Instead: POST /exports creates an export job and returns 202 with a status URL. User polls or refreshes; when done, the download link appears. Slow became invisible.
Backoff in action
A flaky external dependency fails the job. Attempt 1 immediate, attempt 2 after ~30 seconds, attempt 3 after ~2 minutes, then dead-letter. The pattern survives transient blips without amplifying an outage - the difference between resilient and robotic.
Common mistakes
Common mistake
Sending email or generating reports inline in the request handler.
Fix
If any step could take seconds, enqueue it. Respond 202 + status endpoint. The request path owes users speed, not completion.
Common mistake
Jobs failing silently - no statuses, no dead letters, just vanishing work.
Fix
Job states persisted and viewable; exhausted jobs land somewhere a human reviews. Invisible failure is worse than failure.
Working with your AI agent
Require a job contract table before any queue code: job name, payload fields, triggering action, max attempts, backoff curve, dead-letter policy. One worker loop processes all jobs through one runner so logging, retries and statuses stay uniform - per-job bespoke retry logic is how queues become haunted graveyards.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Job Bench
Give the Ledger a real job system: CSV export and nightly summary generation move to background workers with retries, backoff, dead-letter review, and status endpoints - all observable, all restart-safe.
Requirements
- POST creating an export enqueues a job and responds quickly with a status reference - no inline generation
- Worker processes jobs with persisted status transitions: pending, running, done, failed, dead-letter
- Failed jobs retry with growing backoff, then land in dead-letter retaining error details and payload
- Status endpoint shows a job's current state; export completes with a downloadable artifact
- Scheduled job produces the daily summary without any HTTP request triggering it
- Queued jobs survive a worker restart - nothing lost, duplicates handled honestly
Context for your agent
Build a background-job system for my Expense Ledger using a boring local queue (database-table-backed is fine - propose options). Two producers: export endpoint (202 + status URL) and a scheduler tick producing the daily summary. One uniform worker loop with the job contract table we agree on first: payloads validated, bounded attempts, exponential-ish backoff, dead-letter retention with error text. Persist statuses; expose GET /jobs/:id. Simulate a failing job type to exercise retries and dead-lettering end to end.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Audit my Job Bench with suspicion about what happens when things go wrong - because in this system they are supposed to. Verify PASS or FAIL with evidence: (1) interrupt a running job (kill the worker) - does the job recover without duplicating side effects? Report exactly what you observed, (2) force repeated failure - does backoff follow the documented contract and does the job land in dead-letter with usable diagnostics, (3) can user A read user B's job status through any endpoint manipulation, (4) do queued jobs genuinely survive restart? Then tell me: which job, if silently lost right now, would upset users most - and how long until anyone noticed?
Run this against “Background Jobs and Queues” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
Which operation did YOU move out of the request path, and how did users' experience change?
Why does at-least-once delivery mean your handlers must tolerate reruns?
Where do your dead letters go, and who looks at them?
You can now
- Move slow work into queues with honest status visibility
- Configure retries and backoff that ride out transient failures
- Keep background failures visible and diagnosable
Up next
Payments Integration