HTTP/2 significantly improves upon HTTP/1.1 with features like multiplexing and HPACK header compression. HPACK uses a dynamic table that helps shrink data size, making everything faster. For Google Cloud users, it’s key to understand how this dynamic table works, especially with Google Cloud’s load balancers. These load balancers can affect the dynamic table size agreed upon with clients, and any mismatch can slow things down. We’ll explore how this impacts communication and what happens when clients and load balancers aren’t aligned, especially with recent changes to Google Cloud’s HTTP/2 header compression.
HTTP/2 vs HTTP/1: A leap forward in web communication
HTTP/2 significantly speeds up web communication and improves efficiency over its predecessor, HTTP/1.1, with several features that work together to deliver a faster and smoother web experience:
-
Multiplexing allows multiple requests and responses messages to be sent concurrently over a single TCP connection, eliminating the head-of-line blocking that plagued HTTP/1.1.
-
Binary protocol uses binary content instead of texts for messages , making it more efficient to parse, less error-prone, and lighter on the network.
-
HPACK header compression drastically reduces the size of HTTP headers using a sophisticated compression scheme, minimizing overhead and speeding up data transfer. We’ll explore HPACK in detail shortly.
Figure 1: HTTP2 Header Compression. (credit: High Performance Browser Networking, O’Reilly)
-
Server push enables servers to proactively send resources to the client’s cache that it anticipates will be needed, reducing round trips.
-
Stream prioritization allows the client to indicate the importance of different requests, so that critical resources can be delivered faster.
Collectively, these features lead to faster page loads, better resource utilization, and an improved user experience. Now, let’s dive deeper into one of the most impactful of these: HPACK.
What is HPACK? The power behind header compression
HPACK is the specialized compression format for HTTP/2 headers that can reduce header sizes by about 30%. HPACK’s robustness and efficiency are achieved by:
-
Eliminating redundancy: Many headers, like User-Agent or Accept-Language, are repeated across requests. HPACK smartly avoids resending these identical headers in full.
-
Leveraging static and dynamic tables: HPACK employs a sophisticated caching system using two tables, both maintained by the client and the server for each HTTP/2 connection. This system is how HPACK compresses headers and is formally defined in RFC 7541:
-
Static table: This is a predefined, fixed list containing common header fields and common values (e.g., :method: GET, content-type: application/json, scheme: https). Both endpoints know these values in advance, allowing them to send or receive a small, corresponding index number instead of the full header.
-
Dynamic table: Starting empty, this connection-specific table grows as headers are exchanged. When a header field (name and/or value) isn’t in the static table, or if it’s explicitly marked for indexing, it’s added to this dynamic table. Subsequent uses of that same header can then also be referenced by a much smaller index, creating a highly efficient, evolving compression dictionary specific to that connection.
-
-
Employing Huffman encoding: For header values that are new or not found in either table, HPACK uses Huffman encoding. This technique compresses strings more efficiently before they are sent, further reducing the size of the headers.
This combination of these techniques minimizes the overhead associated with HTTP headers, which can be substantial, especially with the increasing complexity of modern web applications and APIs.
Understanding dynamic header table size
As we’ve seen, the dynamic header table is crucial to HPACK’s efficiency. Both the client and the server maintain it, filling it with header fields from exchanged messages to allow for significant compression through indexing.
The size of this dynamic table is fully configurable. HTTP/2 uses the SETTINGS_HEADER_TABLE_SIZE parameter within the SETTINGS frame. This allows each endpoint (client and server) to declare the maximum size of the dynamic table it is willing to support for decoding. The encoder on the sending side then chooses a size up to this advertised limit. By default, the initial size is typically 4096 bytes (4 KiB).
Benefits of larger dynamic header table sizes
Increasing the dynamic header table size, for instance from the common default of 4 KiB up to a larger value like 64 KiB, can offer significant performance benefits, especially in specific scenarios:
-
Improved compression for diverse traffic: The effectiveness of the dynamic table hinges on its ability to store a sufficient variety of relevant headers for the ongoing traffic. If the table is too small for the diversity and volume of headers being sent, frequently used entries might be evicted prematurely (as it operates on a first-in, first-out basis when full). This “churn” significantly reduces the compression ratio, forcing more headers to be sent, in a less compressed form. This issue is particularly pronounced in scenarios like proxies or load balancers that handle many disparate requests over a single connection.
For servers or intermediaries (like load balancers and proxies) that handle numerous different types of requests or requests from many different clients over long-lived connections, a larger table can store a more extensive set of unique header fields. This means more headers can be represented by small index numbers, improving the overall compression ratio. -
Reduced latency : Better compression translates to fewer bytes on the wire. This directly reduces the time it takes to send and receive messages, leading to lower perceived latency for end-users.
-
Increased throughput: By sending headers with less data, more available bandwidth can be utilized for the actual content, potentially increasing overall throughput.
While the default 4 KiB table works well for simple, direct client-to-server connections, environments that multiplex many request streams often benefit from larger tables. It’s a trade-off between memory footprint and compression efficiency.
Dynamic header table size negotiation
For each HTTP/2 connection, the client and the Application Load Balancer must agree on a dynamic header table size. Agreement is mandatory because both the client and load balancer need to maintain the same content in their respective copies of the connection’s dynamic header table. The dynamic header table size, among other parameters, is synchronized by using HTTP/2 SETTINGS frames within the HTTP/2 connection, as described in 6.5.3. Settings Synchronization of RFC 9113. A receiving party for a SETTINGS frame must acknowledge each setting therein, in the order received. For example, if a load balancer sends a SETTINGS frame to a client, the client must acknowledge each setting therein, maintaining the order, before it can send any other frames — including frames with subsequent requests — over the HTTP/2 connection. SETTINGS frames are exchanged at the beginning of an HTTP/2 connection and optionally at any point during the HTTP/2 connection in case any settings updates need to be made.
At the beginning of each new HTTP/2 connection, the client and load balancer start with a default dynamic header table size of 4KiB (4,096 octets), as described in section 4.3.1. ‘Compression State’ of RFC 9113. This default value is considered to be negotiated already.
A load balancer is permitted to send a SETTINGS frame with the SETTINGS_HEADER_TABLE_SIZE setting to give the client the opportunity to select a larger dynamic header table size. The client can choose to reply with a request for a dynamic header table size up to the value of the SETTINGS_HEADER_TABLE_SIZE. Negotiating a larger dynamic table size at the beginning of an HTTP/2 connection reduces the chance that the connection might have to pause data processing in order to negotiate a larger dynamic table size later on in the connection.
The following steps illustrate how a load balancer can offer a larger dynamic header table size, and how a client can take advantage of that offer. In this example, the load balancer offers SETTINGS_HEADER_TABLE_SIZE=64KiB:
-
A new HTTP/2 connection is initialized, and both the client and load balancer understand that the dynamic header table size is 4KiB, because 4KiB is the default value.
-
The load balancer offers a larger dynamic header table size by sending SETTINGS_HEADER_TABLE_SIZE=64KiB in a SETTINGS frame. The negotiated dynamic table size is still 4KiB.
-
The client sends a SETTINGS frame acknowledging that the largest dynamic header table size it could choose to request is 64KiB. The negotiated dynamic table size is still 4KiB.
-
The client sends a frame requesting a dynamic header table size update, up to 64KiB. For example, the client requests 40KiB.
-
After (4) has completed, the client and load balancer have negotiated the new dynamic header table size: 40KiB in this example. At this point, the client can send instructions to set, update, or reference index values within the expanded dynamic header table.
Illustrating the above steps with a flow chart:
The following variations of the above steps are equally valid according to the HTTP/2 RFCs:
Variation 1: The load balancer can choose not to offer a larger dynamic header table size. In this situation only step (1) happens at the beginning of each new HTTP/2 connection.
Variation 2: The client can choose not to request a larger dynamic header table size even though the load balancer has offered a larger dynamic header table size. In this situation, steps (1) through (3) happen at the beginning of each new HTTP/2 connection, step (4) is skipped, and a new dynamic header table size value is not negotiated. The client and load balancer must proceed with the default dynamic header table sizes.
Below is an examination of a packet capture related to Variation 2, during which a client proposes an 8 KB dynamic header table size. Subsequently, the server offers a 64 KB dynamic header table size to the client. The client does not engage in further negotiation but simply acknowledges the server’s communication. Consequently, both the client and the server operate with a 4 KB dynamic header table size at this juncture.
-
Client proposes an 8 KB dynamic header table size
-
Server acknowledges client’s message, and proposes a 64 KB dynamic header table size
-
Client acknowledges server’s message
Latent bug in older http2 client implementations
Specific versions of the apache.hc.core5.http2 HTTP/2 client library contain a latent bug that didn’t present symptoms until Google Cloud classic and global external Application Load Balancers offered support for larger HTTP/2 dynamic header table sizes.
Cause
Affected versions of the apache.hc.core5.http2 HTTP/2 client library contained a bug: This third-party code didn’t follow a valid dynamic header table size synchronization process because it:
-
did not request a dynamic header table size update, thereby skipping step (4) of the valid dynamic header table size synchronization process, and also
-
incorrectly attempted to reference header fields or fields and values in a resized dynamic header table.
Note that (1) by itself isn’t a bug because that results in the Variation 2 valid dynamic header table size synchronization. However, (1) and (2) together produce the bug because they create a situation where the HTTP/2 client proceeds to step (5) when it has not performed step (4).
The effect of the apache.hc.core5.http2 bug was latent until Google rolled out an updated configuration that instructed the first layer Google front ends that power classic and global external Application Load Balancers to offer a 64KiB dynamic header table size.
In a more general sense, when classic and global external Application Load Balancers used default dynamic header table sizes, the apache.hc.core5.http2 bug was latent because the apache.hc.core5.http2 library’s problematic code paths were not executed. The apache.hc.core5.http2 bug only caused symptoms when a load balancer — any load balancer, not just a Google Cloud one — offered a larger dynamic header table size.
The following table summarizes combinations of clients and load balancer behaviors, indicating which combination produces the symptoms of the incident:
Known affected http2 client library versions
The apache.hc.core5.http2 bug was independently identified and fixed by a contributor to the apache.hc.core5.http2 client library. That fix was accepted into the Apache HttpComponents Core project on 2024-10-12. Based on the build date of apache.hc.core5.http2 5.3.1, version 5.3.1 is likely the first version to include this fix
Mitigation and prevention
We recently pushed a temporary configuration to all first layer Google front ends that serve classic and global external Application Load Balancers so that they no longer offer a larger dynamic header table size. Because Classic and global external Application Load Balancers must be RFC-compliant and serve the needs of all customers, this mitigation is temporary. We will eventually offer a 64KiB dynamic header table size.
Please use a version of the apache.hc.core5.http2 client library that includes the fix as described in affected versions.
Conclusion
Dynamic header table size negotiation is a critical but often overlooked aspect of HTTP/2 performance and interoperability. While larger table sizes can improve compression and reduce latency, they require strict adherence to the HPACK synchronization protocol. As highlighted by the apache.hc.core5.http2 bug, any deviation can lead to subtle and hard-to-diagnose failures. To ensure robust and efficient HTTP/2 communication, it’s essential to keep client libraries up to date and validate their behavior in environments that advertise non-default settings. This is particularly important as infrastructure providers continue to optimize their configurations for performance at scale.