Load Balancing and Horizontal Scaling
Stateless services behind balancers, sessions in shared stores, autoscaling driven by honest signals.
Before this: Distributed Systems Thinking
Objective
Run multiple interchangeable instances behind a load balancer: eliminate singleton state, steer traffic with health checks, survive instance death without user impact, and draft an autoscaling policy anchored to symptoms users feel.
Vocabulary
The words to use when directing your agent.
- load balancer
- Traffic director distributing requests across healthy instances.
- horizontal scaling
- Adding more machines instead of enlarging one - capacity by count.
- stateless service
- An instance holding nothing unique - any replica can serve any request.
- health check
- The balancer's probe deciding whether an instance deserves traffic.
- sticky session
- Pinning one user to one instance - usually a smell indicating hidden state.
Understand it
The statelessness audit
Every piece of in-memory state is a scaling landmine: sessions (move to shared store - Level 2 already did), rate-limit counters (shared - Level 3 did), cache (shared or per-instance-with-TTL tolerance), uploaded files (object storage, not local disk). Run the audit honestly: restart an instance and inventory what users lose. Whatever appears on that list is tomorrow's refactor ticket.
Balancers and health truths
Health checks decide traffic: a deep check verifying dependencies beats a shallow ping that happily routes to zombie instances. Draining matters equally - when instances leave the pool (deploy or scale-in), in-flight requests finish gracefully rather than dying mid-response. These mechanics decide whether deploys feel invisible or violent.
Core concepts
Any instance can die at any moment
Horizontal scaling's premise: instances are livestock, not pets. That premise holds ONLY when no instance owns anything unique - no memory-resident sessions, no local file uploads, no in-process rate counters. Statelessness is not an architecture diagram; it is the property that lets a balancer drop one of five instances mid-rush and users never notice.
Autoscale on symptoms, stabilize with cooldowns
Scaling triggers belong on user-feel signals - request latency, queue depth - not vanity CPU graphs that spike during harmless bursts. Add cooldown periods so the fleet breathes instead of flapping up-down-up-down, and remember scale-IN kills instances too: graceful drain before death, or you shed users mid-request.
Examples
Kill one of two
Ledger runs twice behind a local balancer; scripted traffic flows. Kill instance A mid-run: requests in flight drain, new ones route to B, users scroll on unaware. Sessions lived in the shared store, counters were shared - statelessness paid its dividend in a single afternoon demo.
The sticky-session trap
A team 'fixes' login glitches by enabling sticky sessions. Now instances own users, balancers lose flexibility, one hot instance melts while four idle, and every deploy drops logged-in users. Stickiness was the symptom; un-migrated session state was the disease.
Common mistakes
Common mistake
Per-instance in-memory caches or counters assumed global after scaling to two.
Fix
Shared store for correctness-critical state (counters); TTL-tolerant treatment for cacheable state. The L3 lessons generalize: anything 'in-process' is now 'per-instance'.
Common mistake
Autoscaling on CPU alone, flapping the fleet during spiky traffic.
Fix
Symptom-based triggers plus cooldowns. Scaling that whiplash sheds more capacity than it adds.
Working with your AI agent
Before adding instance number two, commission the statelessness audit from your agent: list every in-memory assumption (sessions, counters, caches, temp files) and its migration target. Then have health checks assert DEPENDENCY health, not just liveness - a deep check is the difference between balancing and roulette.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Twin Instances
Run the Ledger twice behind a local load balancer: complete the statelessness migration (sessions, counters, files), prove zero-user-impact instance death with drained traffic, and draft the autoscaling policy you would trust in production.
Requirements
- Two Ledger instances behind one local balancer serving interleaved scripted traffic
- Statelessness audit completed: every singleton state item migrated to shared/equivalent storage
- Health checks verify dependency health; unhealthy instance removed from rotation automatically
- Instance-killed-mid-traffic drill: zero failed user requests, evidence captured
- Graceful drain verified: stopping an instance lets in-flight requests finish
- Autoscaling policy draft: trigger metric, threshold reasoning, cooldowns, scale-in drain - tied to YOUR measured baselines
Context for your agent
Take my Ledger horizontally scalable. First run the statelessness audit together - list every in-memory/per-instance state and migrate each to its shared-home. Stand up two instances behind a local balancer with deep health checks. Coach my kill-drill: terminate one instance under scripted load, prove no user-visible failures and clean drains. Then help me draft the autoscaling policy using MY baseline measurements - thresholds argued from data, cooldowns included, scale-in draining specified.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Break my Twin Instances like a cloud-region outage rehearsal. Attempt each and grade PASS or FAIL: (1) kill an instance mid-request-storm - any user-visible errors? any lost sessions or counters?, (2) degrade a DEPENDENCY on one instance - does it exit rotation before users suffer, or do they hit zombies?, (3) start writing on both instances simultaneously - any counter drift or cache lies emerge?, (4) audit my autoscaling draft: argue the threshold wrong using MY OWN baseline numbers, then propose the correction. Finish by naming the last remaining piece of singleton state, however small, and its blast radius.
Run this against “Load Balancing and Horizontal Scaling” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
List what your statelessness audit found and where each item moved.
Why do sticky sessions signal unfinished architecture?
Which metric did you anchor autoscaling to, and what real number justifies it?
You can now
- Scale horizontally by eliminating singleton state, not adding hardware prayers
- Steer and drain traffic with health checks that tell the truth
- Draft autoscaling policies from measured baselines, not folklore
Up next
Fault Tolerance and Disaster Recovery