Capacity Planning for Peak Traffic: Lessons From Black Friday Incidents

Peak traffic events don’t cause outages because teams didn’t work hard. They cause outages because “more traffic” hides a dozen different failure modes behind one deceptively simple word.

Every year, retailers, ticketing platforms, and any business with a predictable seasonal spike relearn the same lesson: the system that comfortably handles 10x normal traffic on a load test can still fall over at 4x normal traffic in production, on the day it actually matters. The gap between “we load tested this” and “it survived Black Friday” is where capacity planning either earns its keep or reveals itself as theater.

Traffic Growth Is Never Linear Across Your Stack

The first mistake in capacity planning is treating “10x traffic” as a single number to plan against. In reality, a 10x increase in top-of-funnel traffic (homepage visits, product page views) rarely translates to a uniform 10x increase everywhere downstream. Some components see far more than 10x, some see far less, and a few see a completely different shape of load, not just more of the same shape.

Consider a typical e-commerce flow during a flash sale: homepage traffic might increase 8x, but add-to-cart events might spike 15x because a much higher fraction of visitors are actively trying to buy (versus browsing), and checkout attempts might spike 20x in the first ten minutes before settling to 6x for the rest of the day, because a flash sale creates an initial burst of simultaneous demand that a steady 10x traffic multiplier completely fails to capture. Inventory-check calls, meanwhile, might spike disproportionately higher than checkout itself, because impatient users refresh product pages repeatedly to see if a sold-out item has restocked — a behavior pattern that doesn’t exist at normal traffic levels at all.

This means capacity planning has to be done per-component, using traffic ratios specific to that component’s actual behavior under peak conditions, not a single blanket multiplier applied uniformly across the architecture diagram. The components most likely to be under-provisioned are exactly the ones whose peak-to-normal ratio differs most from the overall traffic ratio — and those are rarely the components anyone thinks to double-check, because “10x more visitors, so I’ll provision 10x” feels intuitively complete.

Load Testing Lies When It Doesn’t Match Real Traffic Shape

The second mistake is load testing with a traffic pattern that doesn’t resemble how real peak traffic actually arrives. A load test that ramps smoothly from baseline to 10x over thirty minutes and holds steady is testing a completely different failure mode than real flash-sale traffic, which typically arrives as a near-instantaneous step function — baseline one second, 12x baseline the next, sustained for two hours, then a slower decay.

The step-function arrival pattern stresses systems in ways a gradual ramp never will: connection pools that scale gracefully under gradual growth can be instantly exhausted by a sudden step, autoscalers that would have had time to provision new capacity during a gradual ramp are caught flat-footed by a step, and caches that would have warmed naturally during a ramp start completely cold at the exact moment they’re needed most.

Realistic load testing for peak events needs to specifically include:

• A sudden step-function ramp, not just a gradual one, to validate behavior under the arrival pattern peak events actually produce.
• Sustained load for the actual expected duration of the peak, not just a brief spike — many failure modes (memory leaks, slow resource leaks in connection pools, log storage filling up) only manifest after sustained load, not immediately at the moment of the spike.
• Realistic request mix, reflecting the actual disproportionate increase in specific actions (cart adds, inventory checks, checkout attempts) rather than a uniform scale-up of “normal day” traffic patterns.
• Cold-cache conditions at the start of the test, since a real flash sale often starts at a scheduled time when caches haven’t yet been warmed by the specific surge of interest in whatever’s on sale.

A load test that skips any of these dimensions can pass cleanly while still leaving your system completely unprepared for the actual event, because it validated a traffic pattern that will never actually occur.

The Database Is Almost Always the Real Bottleneck

Stateless application tiers are, relatively speaking, the easy part of peak capacity planning — horizontal scaling (more replicas behind a load balancer) is a well-understood, largely mechanical problem, assuming your autoscaling and node provisioning are configured correctly. The database tier is where peak traffic incidents actually originate more often than anywhere else, because databases don’t scale horizontally as easily, and connection limits, lock contention, and I/O throughput all have hard ceilings that application-tier autoscaling cannot paper over.

A few specific, recurring failure patterns worth planning against explicitly:

• Connection pool exhaustion. If each application replica opens its own pool of database connections, and your autoscaler adds replicas rapidly in response to traffic, you can hit the database’s maximum connection limit purely from connection count, independent of query load — every additional replica is additional overhead even before it serves a single request. A connection pooler (like PgBouncer for Postgres) sitting between the application tier and the database, capping total connections regardless of application replica count, is close to mandatory for any peak event where autoscaling is expected to add significant replica count.
• Read replica lag under write-heavy bursts. A flash sale generates a disproportionate burst of writes (orders, inventory decrements) compared to a normal day. If your read replicas fall meaningfully behind during this burst, and your application reads inventory counts from a replica, you can sell items that are actually already sold out, or show incorrect availability — a correctness failure, not just a performance one, that’s specific to peak write-heavy conditions and won’t show up in normal-day testing.
• Row-level lock contention on hot rows. Inventory decrement operations on a small number of extremely popular items (the specific flash-sale item everyone wants) can create severe lock contention on those specific database rows, even when the database’s overall load looks completely healthy in aggregate metrics. This is a pattern that’s invisible in average-case load testing and only appears when testing against the actual, specific hot-item access pattern a real sale produces.

Feature Flags as a Load-Shedding Mechanism

The most resilient peak-traffic architectures don’t just try to handle 100% of expected load flawlessly — they build in explicit, pre-planned degradation paths that trade some functionality for guaranteed availability of the core revenue path. This is where feature flags stop being just a deployment tool and become a genuine capacity management tool.

Concrete examples that mature e-commerce platforms build in advance of peak events:

• Disable personalized recommendations (“customers who bought this also bought…”) during peak load, since these are usually expensive database or ML-service calls that add latency without being essential to completing a purchase.
• Simplify or cache search results more aggressively, accepting slightly less fresh or less personalized results in exchange for dramatically reduced database load.
• Queue checkout requests explicitly with a visible “you’re in line” experience, rather than letting checkout attempts pile up invisibly against an overloaded payment gateway until they start timing out unpredictably. A visible, honest queue is a dramatically better user experience than silent failures, and it gives the backend a controlled admission rate instead of an uncontrolled flood.
• Reduce non-critical background job frequency (analytics batch jobs, report generation, non-urgent email sends) during the peak window, freeing up shared infrastructure capacity for the customer-facing critical path.

Building these degradation paths, and rehearsing the decision of when to flip them, before the peak event — not improvising them live during an active incident — is what separates teams that gracefully shed non-essential load from teams that experience an uncontrolled, total outage because everything failed together, all at once, with no pre-agreed fallback.

CDN and Edge Caching as a First Line of Defense

Before any request reaches your origin infrastructure at all, a properly configured CDN and edge caching layer can absorb a substantial fraction of peak traffic, particularly for read-heavy pages like product listings and category pages that don’t need to be freshly rendered for every single visitor. Extending cache TTLs deliberately during a known peak window, even at the cost of slightly staler content for non-critical pages, shifts load away from origin servers and databases exactly when they can least afford additional demand. This is often one of the cheapest, highest-leverage capacity levers available, since it requires no additional infrastructure spend, only a deliberate, pre-planned change to caching configuration ahead of the event.

The Runbook Rehearsal

Capacity planning that stops at infrastructure provisioning and load testing is only half the job. The other half is making sure humans know what to do when, despite all the preparation, something still goes wrong during the actual event — because something almost always does, at some scale, somewhere in the stack.

This means, in the weeks before a known peak event:

1. Identify the specific on-call team for the event window, distinct from (and usually larger than) the normal rotation, with clear escalation paths that don’t depend on any single individual being reachable.
2. Pre-stage the load-shedding feature flags discussed above, tested in advance, so flipping them during a real incident is a rehearsed one-click action, not a risky, improvised code change under pressure.
3. Set explicit, pre-agreed thresholds for each degradation step — at what error rate or latency does the team disable recommendations, at what point does the checkout queue activate — so the decision during the event is a lookup against an agreed plan, not a stressful real-time debate.
4. Run a game day specifically simulating the peak event, including simulating the decision to activate degradation modes, so the team has practiced the actual sequence of actions, not just discussed it in the abstract.

Third-Party Dependencies Have Their Own Peak Limits

It’s easy for capacity planning to focus entirely on infrastructure you directly control and quietly assume that third-party dependencies, such as payment gateways, tax calculation services, shipping rate APIs, and fraud scoring providers, will simply scale to meet whatever load you send them. In practice, most third-party providers have their own rate limits and their own capacity constraints, and are often experiencing their own peak-traffic incidents at the same time as yours, since a shared payment processor is typically serving many retailers simultaneously during the same industry-wide peak shopping events.

Capacity planning for a peak event needs to explicitly include a conversation with every critical third-party dependency: what rate limits apply to your account, whether those limits can be temporarily raised for a known peak window, and what their own historical performance and incident rate has looked like during past comparable events. A checkout flow that scales flawlessly on your own infrastructure but silently starts failing because a fraud-scoring API enforces a hard rate limit you never asked about is a completely preventable outage, discovered the hard way instead of in a planning conversation weeks earlier.

Equally important is deciding, in advance, what your system does when a third-party dependency degrades or hits its own limits during your peak. A graceful, pre-built fallback, such as queuing requests, temporarily skipping optional fraud checks under clearly defined risk tolerances, or degrading gracefully to manual review, prevents a partner’s incident from becoming your own outage, whereas no fallback plan guarantees it will.

After the Event: The Retro That Actually Matters

The most valuable capacity planning artifact isn’t produced before the peak event — it’s produced immediately after, while the specific numbers are still fresh: actual traffic multipliers per component, compared against the multipliers assumed during planning; which degradation flags were actually flipped, and how much load they actually shed; which alerts fired usefully and which fired as noise during the chaos of the event itself.

This retro data is what makes next year’s capacity plan more accurate than this year’s guess. Teams that treat peak-event capacity planning as a fresh exercise every year, without systematically incorporating the previous year’s actual measured ratios, are re-deriving the same lessons from scratch — often the hard way, during the event itself, instead of the easy way, from last year’s data.