Skip to content
Part 10SecurityChapter 9 of 14

DDoS Defense Architecture ​

TL;DR ​

A distributed denial-of-service attack does not break a control; it spends a resource faster than the defender can supply it. The resource may be bandwidth on a link, entries in a connection table, CPU for TLS handshakes, or database time behind one expensive endpoint. Defense is therefore a capacity and placement problem before it is a detection problem: each layer must discard hostile work using less of its own resource than the attacker spent to send it, and must do so before the work reaches a layer where it is more expensive.

That principle produces the standard architecture. Volumetric floods are absorbed at a distributed, anycast edge with far more bandwidth than any single site. Transport-layer state exhaustion is defeated by refusing to allocate state for an unverified peer. Application-layer floods are met with identity-based limits, cost-aware admission, and challenges, because each request is individually valid. The origin is reachable only through the edge, or the whole arrangement is bypassed. Rate Limiting owns limiter algorithms and distributed budgets, and API Threat Boundaries and Abuse Resistance owns authenticated abuse. The subject here is surviving traffic whose purpose is exhaustion.

Primary Evidence and Scope ​

SourceWhat it establishesBoundary
RFC 2827 (BCP 38)Ingress filtering at network edges prevents spoofed source addresses, the precondition for reflection attacksWorks only where operators deploy it
RFC 4987SYN flooding mechanics and mitigations including SYN caches and SYN cookiesTCP-specific
Rossow, Amplification Hell (NDSS 2014)14 UDP-based protocols usable for amplification, with bandwidth amplification factors up to 4,670Survey of protocols as deployed at the time
GitHub incident report (2018)A memcached reflection attack peaking at 1.35 Tbps and 126.9 million packets per second, mitigated by diverting traffic to a scrubbing providerOne incident
Antonakakis et al., Mirai (USENIX Security 2017)An IoT botnet that peaked near 600,000 infected devices and its attack historyOne botnet family
Cloudflare and Google reports on HTTP/2 Rapid Reset (2023)Request floods of 201 million and 398 million requests per second from a protocol feature that cancels streams cheaplyVendor-reported peaks for one vulnerability
RFC 9000QUIC limits a server to sending three times the bytes received from an unvalidated addressThe protocol's anti-amplification rule

Sizing examples below are illustrative.

Attack Classes by Exhausted Resource ​

Classify an attack by what it consumes, because that determines where it must be stopped.

ClassResource exhaustedTypical formWhere it must be stopped
VolumetricLink and transit bandwidth, packet-forwarding capacityReflection and amplification over UDP, direct floods from botnetsUpstream of the site, across many locations
Protocol or stateConnection tables, half-open queues, load-balancer and firewall stateSYN floods, fragment floods, connection exhaustionThe first device that would allocate state
ApplicationCPU, memory, worker pools, database and downstream capacityHTTP request floods, slow requests, expensive queries, protocol abuse such as rapid stream resetsThe first layer that can attribute and price a request

Two properties distinguish the classes. Volumetric attacks are mostly spoofed and stateless, so they can be filtered by packet properties without understanding sessions. Application attacks are unspoofed and well-formed: every request completes a handshake and passes validation, so discarding them requires deciding which clients to believe.

Why Amplification Works ​

Reflection needs two things: a network that lets the attacker send packets with the victim's source address, and a public service that answers a small request with a large response. The attacker's cost is the request; the victim receives the response.

Documented, Rossow: the study measured bandwidth amplification factors across 14 protocols, up to 4,670. Documented, GitHub: the 2018 attack used exposed memcached servers, whose amplification the report describes as up to 51,000 times. An attacker controlling 100 Mbps of spoofed traffic and an amplifier with a factor of 500 delivers 50 Gbps, more than most single sites buy.

Source-address filtering at network edges removes the precondition, which is why it is a published best current practice. A defender cannot rely on other networks having deployed it.

The Layered Defense Path ​

Each stage is cheaper per discarded unit than the one after it. A packet dropped by a stateless filter costs nanoseconds. A connection refused by a SYN cookie costs one hash. A request rejected after TLS has already cost a handshake. A request rejected by the application has cost a worker, a parse, and possibly a database query. The design goal is to make the earliest possible stage do the discarding.

Absorbing Volume at the Edge ​

Bandwidth attacks are won by having more capacity than the attacker in more places than the attacker can concentrate on.

Anycast announces the same address prefix from many locations, so the routing system delivers each packet to a nearby one. An attack sourced from machines around the world is divided among sites along the same boundaries as legitimate traffic, and no single site receives the total. The architecture is described in CDN Architecture and Cloudflare System Design.

Three operating models exist:

  • Always-on proxy. All traffic enters through a provider's edge. No detection delay and no routing change during an attack; the provider is in the path of every request, with the latency and dependency that implies.
  • On-demand diversion. Under attack, routes are changed so traffic passes through scrubbing centers and clean traffic is returned over a tunnel. The site is exposed during detection and route convergence. Documented, GitHub: the 2018 report describes exactly this sequence, with the site unavailable for several minutes before diversion took effect.
  • Remotely triggered black hole. The upstream network drops all traffic to the attacked address. This completes the attacker's goal for that address in exchange for saving everything else on the network. It is a last resort and a reason to keep unrelated services on separate addresses.

At the edge, stateless filters remove what no legitimate client sends: UDP to an address that serves only TCP, fragments, source ports of known amplifiers, and packets for protocols the service does not speak. Operators distribute such rules to routers at the network border so the drop happens before the traffic is carried inward.

Refusing to Hold State ​

A TCP server traditionally allocates a record when it receives a SYN and holds it until the handshake completes or times out. A SYN flood sends handshakes that never complete, filling that table so genuine connections are refused.

Documented, RFC 4987: SYN cookies remove the allocation. The server encodes the connection parameters into the initial sequence number of its SYN-ACK and keeps nothing. A genuine client returns that number in its ACK, from which the server reconstructs the state. A spoofed source never replies, and the server spent one hash computation.

The same idea generalizes and should be applied everywhere a peer is unverified:

  • Do not send more than you received. Documented, RFC 9000: before validating a client's address, a QUIC server may send at most three times the bytes it has received, which caps its usefulness as an amplifier.
  • Prove the return path first. Retry tokens and cookies force the client to demonstrate it can receive packets at its claimed address before the server commits resources.
  • Keep middleboxes stateless where possible. A stateful firewall or load balancer in front of servers is itself a connection table to exhaust. Designs that hash packets to backends consistently, without per-connection memory on the forwarding path, do not have that failure mode. See Load Balancing.

The transport details, including handshake costs and kernel queues, are in Network Transport Internals.

Application-Layer Floods ​

When every request is valid, filtering becomes a judgment about clients, and the attacker's advantage is cost asymmetry: a request that takes microseconds to send may take the defender a database query, a template render, or a password hash.

Documented, 2023 Rapid Reset reports: attackers opened HTTP/2 streams and cancelled them immediately. Cancellation freed the client's concurrency slot while the server had already begun work, so the protocol's concurrent-stream limit did not bound the request rate. Cloudflare reported a peak of 201 million requests per second and Google 398 million, with Cloudflare noting the botnet involved was on the order of 20,000 machines. The lesson generalizes: any operation that is cheap to start and cheap to abandon for the client, but not for the server, is an amplifier.

The defensive toolkit, in rough order of application:

  1. Serve from cache. A cached response costs the edge almost nothing. Attackers therefore target what cannot be cached: search, login, cart, and query strings that defeat cache keys. Normalize cache keys and refuse unbounded variation.
  2. Limit by identity, at several granularities. Per address, per network prefix, per authenticated account, per API key, per session, and per TLS or client fingerprint. A botnet of 20,000 machines sending 50 requests per second each produces one million requests per second while every individual address looks moderate, so per-address limits alone do not see it.
  3. Price requests by cost, not count. Give each endpoint a weight reflecting its backend cost and charge budgets in those units. Ten search queries should not be treated like ten static fetches.
  4. Challenge when uncertain. A computational or interactive challenge raises the attacker's per-request cost and separates automated clients from browsers. It also harms legitimate automation and accessibility, so apply it selectively and only under suspicion.
  5. Bound connection behavior. Limit streams opened and reset per connection, header and body read times, and request size. Close connections that misbehave instead of only rejecting requests.
  6. Shed by priority at the origin. When load still exceeds capacity, serve authenticated and in-progress sessions first and fail the rest quickly. See Backpressure.

Origin and Dependency Protection ​

An edge that absorbs attacks is worth nothing if the origin can be addressed directly. Attackers find origin addresses in historical DNS records, certificate transparency logs, outbound mail headers, and misconfigured subdomains.

  • Accept origin connections only from the edge, enforced by network allowlists and by mutual TLS or a signed header the origin verifies.
  • Rotate the origin address after any exposure, and never publish it in a DNS record.
  • Keep administrative and deployment access on a separate path, so mitigation can be changed while the public path is saturated.

Dependencies are attack surface too. Authoritative DNS, an identity provider, a payment gateway, or a shared rate-limit store can be the real target, and an attack on them looks like an application outage. The 2016 attack on a managed DNS provider made many unrelated sites unreachable without touching their servers. Use more than one DNS provider where the product justifies it, and see DNS and Connection Management.

For multi-tenant platforms, the blast radius is the question: one attacked customer must not exhaust the capacity shared with others. Assigning tenants to overlapping subsets of edge capacity limits the damage to the tenants sharing that exact subset, as described in Cell-Based Architecture and Shuffle Sharding.

Capacity and Cost Model ​

All inputs are illustrative.

SYN flood. Ten million SYN packets per second at 60 bytes each:

text
bandwidth        = 10,000,000 x 60 x 8      = 4.8 Gbit/s
state if stored  = 10,000,000 x 60 s hold   = 600 million half-open entries
state with cookies                          = 0

The bandwidth is survivable for a well-connected site. The state is not, which is why the mitigation is to store none.

Application flood against a small origin. An origin sustains 20,000 requests per second at its latency objective. A botnet sends one million.

text
required discard rate at the edge  = 1 - 20,000 / 1,000,000  = 98%

If discrimination is imperfect and 5% of hostile requests pass, the origin receives 50,000 hostile requests per second, more than twice its capacity, before any legitimate traffic. Edge filtering alone is insufficient; the origin still needs admission control and a cheap failure mode.

Autoscaling as an attack outcome. If each instance costs 0.20 currency units per hour and serves 1,000 requests per second, absorbing one million requests per second by scaling out costs 200 per hour for as long as the attacker continues. Scaling limits and budget alarms are part of the defense. A capped service that sheds is often the correct design; an uncapped one converts denial of service into denial of wallet.

Failure Traces ​

The origin was never hidden ​

A site sits behind a proxy service. Its origin address appears in an old DNS record for a staging hostname. The attacker sends a volumetric flood directly to the origin's hosting provider, which black-holes the address. The edge is healthy and serves errors. Lock the origin to edge sources and treat any address disclosure as requiring rotation.

Retries amplify the attack ​

The edge sheds 30% of requests with an error. Mobile clients retry immediately, three times each. Legitimate load nearly doubles on top of the attack, and the service that was degraded becomes unavailable. Return an explicit retry delay, make clients honor it with jitter, and budget retries as described in Retries, Timeouts, and Hedging.

One expensive endpoint ​

A product search endpoint runs an unindexed query taking 400 milliseconds of database time. Fifty requests per second saturate the database, and the request rate never crosses any per-address or global limit. Find such endpoints before attackers do by ranking endpoints by backend cost per request, and give them their own small budgets.

The mitigation depended on the attacked path ​

Operators need to push a new filter rule. The configuration interface is served through the same edge and the same authentication service that is under attack, and the login times out. Keep an out-of-band management path and pre-approved emergency rules that can be enabled without the primary control plane.

Legitimate users behind one address ​

A per-address limit of 100 requests per second blocks a university and a mobile carrier whose users share a small set of translated addresses. Support volume becomes the second incident. Prefer authenticated identity where it exists, widen limits for known shared networks, and degrade to challenges before blocking.

Operations and Verification ​

Maintain a runbook with named decisions: who may enable challenges site-wide, who may request a black hole, and what the communication path is when primary tools are down. The command structure is in Incident Command and Learning.

Observe at every layer separately: packets and bits per second by protocol at the edge, SYN and connection-table occupancy, handshake rate, requests per second by endpoint cost class, cache hit ratio, challenge issue and pass rates, origin admission rejections, and dependency saturation. A single request-rate graph hides which layer is failing.

Test deliberately and with authorization from every network involved. Load-test the origin's shedding behavior, verify that the origin refuses direct connections, exercise route diversion on a schedule so that convergence time is known, and replay captured attack patterns against a staging edge. An untested mitigation is a hypothesis.

Decision Framework ​

SituationChoiceReason
Public web property with no tolerance for detection delayAlways-on anycast proxy with origin lockdownNo exposure window; volume is absorbed where capacity exists
Non-HTTP service or own address space, attacks are rareOn-demand route diversion to scrubbingPreserves direct paths in peacetime; accept minutes of exposure
TCP service on your own hostsSYN cookies enabled, stateless forwarding in frontRemoves the half-open table as a target
UDP service you operateAddress validation before large responses; response no larger than a small multiple of the requestDo not become someone else's amplifier
API with authenticated clientsPer-key cost budgets, tight limits for unauthenticated routesIdentity is a better limiter key than address
Endpoints with high backend costSeparate small budgets, caching or precomputation, challenge on anomalyThese are where a small flood succeeds
Multi-tenant platformPer-tenant capacity isolation and shuffle shardingOne tenant's attacker must not become every tenant's outage
Cost-sensitive service with autoscalingHard scaling caps and shed-on-overloadBound the bill as well as the latency

Finally, ask of each layer: what resource does it spend per hostile unit, what does the attacker spend, and which earlier layer could have discarded it?

References ​


A practical reference for distributed system design. Released under the MIT License.