A newly identified denial-of-service vulnerability in OpenSSL, dubbed “HollowByte” by its discoverers at Okta, has been patched discreetly, leaving many servers potentially exposed. The flaw allows attackers to consume significant amounts of server memory with minimal data, resulting in resource exhaustion. This issue surfaced when Okta’s Red Team reported details of the bug, which OpenSSL addressed in June releases without issuing a CVE, advisory, or explicit changelog entry. The affected OpenSSL versions include 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, with all prior releases on these branches also susceptible. The lack of formal notification means that standard security scanning tools may not identify vulnerable systems, potentially leaving them open to exploitation.
The HollowByte vulnerability stems from a fundamental flaw in how older OpenSSL versions handled TLS handshake messages. Specifically, prior to the validation of message content, the system would allocate a receive buffer based on a declared length in the message’s 4-byte header. Even an 11-byte message with a declared body length of up to 131 KB could trigger this premature allocation. Once the buffer was established, the worker thread would block, waiting for data that would never arrive, effectively rendering the connection useless and consuming resources without any successful authentication or key exchange.
The Unreturning Memory Footprint of HollowByte
While a simple connection exhaustion attack is not new, HollowByte’s persistence is amplified by its interaction with glibc, a common C standard library on Linux systems. After an attack, OpenSSL might free the allocated buffer, but glibc’s memory management typically holds onto small and medium-sized chunks for potential reuse rather than returning them immediately to the operating system kernel. Okta’s testing revealed that by varying the claimed message size in each connection, attackers could prevent glibc from effectively reusing freed memory. This leads to heap fragmentation, a persistent increase in resident set size, and a lasting impact on server performance even after the attacker has disconnected.
In practical terms, Okta’s Red Team observed significant memory lock-up on test systems. A 1 GB server running NGINX was reportedly rendered out-of-memory (OOM) after accumulating 547 MB of frozen memory fragments. On a larger 16 GB server, the HollowByte attack consumed 25% of the system’s total memory, even without hitting the theoretical connection limit. This behavior led the Red Team to conclude that “standard connection-limiting defenses won’t stop it,” as the issue lies in the core memory handling rather than the number of concurrent connections.
Okta has not publicly released exploit code, and as of July 18, no public proof-of-concept was found on platforms like GitHub. However, the implications for server administrators are clear: unpatched systems are vulnerable to a stealthy resource depletion attack that bypasses conventional denial-of-service mitigation strategies.
OpenSSL’s Classification of the HollowByte Flaw
A significant point of contention is OpenSSL’s decision to classify HollowByte as a “bug or hardening” fix rather than a vulnerability requiring a CVE and public advisory. The patch itself, submitted by Matt Caswell, clearly indicates a fix for a memory allocation issue. OpenSSL’s internal security policy categorizes issues into severity tiers from Critical to Low, and the “bug or hardening” designation falls outside these classifications. Typically, even Low-severity issues are accompanied by a CVE identifier, a changelog entry, and a listing on the project’s vulnerabilities page.
The reasoning behind this classification remains unstated by OpenSSL. One potential argument could be that the allocated memory per connection (131 KB) is relatively small, and all TLS servers allocate memory per connection. However, Okta counters that the critical aspect is that this memory is not returned. In January, OpenSSL did assign CVE-2025-66199, a Low-severity bug involving TLS 1.3 certificate compression, where a peer-supplied length could grow a heap buffer before validation, consuming approximately 22 MiB per connection. This earlier issue, though requiring specific configuration prerequisites, received a CVE, unlike HollowByte which needs no special conditions.
The same June 9 release that contained the HollowByte fix also addressed CVE-2026-34183, a Moderate-severity issue involving unbounded memory growth in the QUIC PATH_CHALLENGE handler. Both of these were memory-exhaustion denial-of-service flaws and were formally documented. In contrast, HollowByte’s lack of a CVE creates challenges for downstream distributions. Red Hat, for instance, defaults to backporting security fixes rather than updating the package version. Without a CVE to index against, it becomes difficult for administrators to verify if a patched package on their system actually incorporates the HollowByte fix, as the displayed version may not reflect the internal changes.
To confirm the inclusion of the HollowByte fix, system administrators would ideally consult the package changelog for mentions of pull request 30792 (for master and 4.0), 30793 (for 3.6, 3.5, and 3.4), or 30794 (for 3.0), or directly query their distribution’s maintainers. For those who compile OpenSSL from source, upgrading to the patched releases and restarting any affected services is the recommended course of action.
It is important to note that the HollowByte fix is currently limited to TLS connections. OpenSSL has stated that addressing DTLS would require more significant changes and has not committed to a timeline for its inclusion. A review of the source code for the 3.6.2 and 3.6.3 tags indicates that the DTLS handshake file remains byte-identical concerning the vulnerability. In the latest 4.0.1 release, this path still allows the buffer size to be dictated by the peer’s declared length, leaving DTLS implementations potentially exposed. OpenSSL has yet to formally classify this DTLS path or provide a commitment to fixing it, and current public documentation remains silent on the matter.
The lack of clear communication and documentation around the HollowByte fix leaves a degree of uncertainty regarding its full implementation and impact. The next steps will likely involve monitoring OpenSSL for any further statements or updates regarding the DTLS aspect of this issue, and for downstream distributions to provide clearer guidance on how administrators can verify the patch’s application to their systems. Organizations relying on OpenSSL should proactively investigate their environments for the presence of this vulnerability, especially if they are not on the latest patch levels or have not received specific advisories from their vendors.