Most engineering teams rely on auto-scaling to save them. But at the “Super Bowl standard” of scale, auto-scaling is a lie. It is too reactive. By the time your cloud provider spins up new instances, your latency has already spiked, your database connection pool is exhausted and your users are staring at a 500 error.
Here are the four architectural patterns we use to survive massive concurrency. These apply whether you are streaming touchdowns or processing checkout queues for a limited-edition sneaker drop.
1. Aggressive load shedding
The biggest mistake engineers make is trying to process every request that hits the load balancer. In a high-concurrency event, this is suicide. If your system capacity is 100,000 requests per second (RPS) and you receive 120,000 RPS, trying to serve everyone usually results in the database locking up and zero people getting served.
