OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests

Okta’s Red Team has disclosed a denial-of-service vulnerability in OpenSSL, dubbed HollowByte, that allows an attacker to freeze server memory using 11-byte TLS requests. The flaw affects OpenSSL versions prior to the June 9 releases: 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The vulnerability lies in how OpenSSL handles TLS handshake messages: it trusts the declared body length in the 4-byte header, allocating up to 131 KB of memory before receiving any actual body data. When the connection is dropped, glibc’s memory allocator fragments the heap, preventing memory reuse and causing resident set size to climb. In Okta’s tests, a 1 GB server was OOM-killed with 547 MB of frozen memory, and a 16 GB server lost 25% of system memory. OpenSSL’s security team classified this as a ‘bug or hardening’ fix, not a vulnerability, and did not assign a CVE, advisory, or changelog entry. The fix covers TLS only; DTLS remains unpatched. Downstream distributions like Red Hat may be affected due to backporting practices. Administrators should upgrade to the fixed releases or apply the relevant pull requests (30792-30794).