A vulnerability in the HTTP/2 protocol dubbed “Rapid Reset” has led to record DDoS attacks on web servers in recent months. Google, AWS and Cloudflare jointly revealed the attacks and vulnerability today, but noted that every modern web server remains vulnerable to the attack technique. Web server vendors and projects also announced mitigation measures and patch plans.
Google said the attacks peaked at 398 million requests per second (rps) — more than five times larger than the previous record set in February 2023 — and more web traffic in two minutes than Wikipedia received in the entire month of September. Cloudflare said it saw attacks peak just above 201 million requests per second.
Google, AWS and Cloudflare said they were able to limit damage from the attacks. “While initially we saw some impact to customer traffic — affecting roughly 1% of requests during the initial wave of attacks — today we’ve been able to refine our mitigation methods to stop the attack for any Cloudflare customer without it impacting our systems,” Cloudflare said.
One troubling fact is that the attackers were able to generate the attack with a botnet of just 20,000 machines. “There are botnets today that are made up of hundreds of thousands or millions of machines,” Cloudflare said in a technical blog post on the vulnerability (CVE-2023-44487). “Given that the entire web typically sees only between 1–3 billion requests per second, it’s not inconceivable that using this method could focus an entire web’s worth of requests on a small number of targets.”
Equally troubling is how widespread the vulnerability is.
Also read: How to Stop DDoS Attacks in Three Stages
Featured Partners
‘Every Modern Web Server’ Affected
Cloudflare noted that because the attack abuses an underlying weakness in the HTTP/2 protocol, “we believe any vendor that has implemented HTTP/2 will be subject to the attack. This included every modern web server.”
“We, along with Google and AWS, have disclosed the attack method to web server vendors who we expect will implement patches. In the meantime, the best defense is using a DDoS mitigation service like Cloudflare’s in front of any web-facing web or API server.”
Web server vendors and open source projects, including Apache Tomcat, Microsoft and several others, issued guidance for dealing with the vulnerability; the growing number of announcements can be found in the CVE listing.
NGINX, for example, recommended a number of configuration changes to minimize the attack surface:
- keepalive_requests should be kept at the default setting of 1000 requests
- http2_max_concurrent_streams should be kept at the default setting of 128 streams
- limit_conn enforces a limit on the number of connections allowed from a single client and should be added “with a reasonable setting balancing application performance and security”
- limit_req enforces a limit on the number of requests that will be processed within a given amount of time from a single client, and should also balance application performance and security.
NGINX said it will issue a patch tomorrow that will impose a limit on the number of new streams that can be introduced within one event loop. The limit will be set at twice the value configured using the http2_max_concurrent_streams directive. “The limit will be applied even if the maximum threshold is never reached, like when streams are reset right after sending the request (as in the case of this attack),” NGINX said.
Also read:
- How to Prevent DDoS Attacks: 5 Steps for DDoS Prevention
- How To Tell If You’ve Been DDoSed: 5 Signs of a DDoS Attack
How the HTTP/2 ‘Rapid Reset’ Attack Works
In a technical blog post on the HTTP/2 “Rapid Reset” attack, Google noted that “A primary design goal of HTTP/2 was efficiency, and unfortunately the features that make HTTP/2 more efficient for legitimate clients can also be used to make DDoS attacks more efficient.”
In essence, the Rapid Reset attack works by abusing an HTTP/2 feature called “stream cancellation” by repeatedly sending requests and then immediately canceling them.
The HTTP/2 protocol lets clients indicate to a server that a previous stream should be canceled by sending a RST_STREAM frame, Google noted. The protocol does not require the client and server to coordinate the cancellation, and the client may do so unilaterally. The client may also assume that the cancellation will take effect immediately when the server receives the RST_STREAM frame, before any other data from that TCP connection is processed.
“This attack is called Rapid Reset because it relies on the ability for an endpoint to send a RST_STREAM frame immediately after sending a request frame, which makes the other endpoint start working and then rapidly resets the request,” Google said. “The request is canceled, but leaves the HTTP/2 connection open.”
The HTTP/2 Rapid Reset attack built on this capability is simple, Google said:
“The client opens a large number of streams at once as in the standard HTTP/2 attack, but rather than waiting for a response to each request stream from the server or proxy, the client cancels each request immediately. The ability to reset streams immediately allows each connection to have an indefinite number of requests in flight. By explicitly canceling the requests, the attacker never exceeds the limit on the number of concurrent open streams. The number of in-flight requests is no longer dependent on the round-trip time (RTT), but only on the available network bandwidth.”
In a typical HTTP/2 server implementation, the server “will still have to do significant amounts of work for canceled requests, such as allocating new stream data structures, parsing the query and doing header decompression, and mapping the URL to a resource,” Google said.
For reverse proxy implementations, “the request may be proxied to the backend server before the RST_STREAM frame is processed. The client on the other hand paid almost no costs for sending the requests. This creates an exploitable cost asymmetry between the server and the client. Another advantage the attacker gains is that the explicit cancellation of requests immediately after creation means that a reverse proxy server won’t send a response to any of the requests.”
Mitigations can take multiple forms, but mainly center around tracking connection statistics and using signals and business logic to determine how useful each connection is, Google said. “For example, if a connection has more than 100 requests with more than 50% of the given requests canceled, it could be a candidate for a mitigation response. The magnitude and type of response depends on the risk to each platform, but responses can range from forceful GOAWAY frames as discussed before to closing the TCP connection immediately.
“To mitigate against the non-cancelling variant of this attack, we recommend that HTTP/2 servers should close connections that exceed the concurrent stream limit. This can be either immediately or after some small number of repeat offenses.”
Read next:
