Preface: There are two types of buffer overflows: stack-based and heap-based. Heap-based, which are difficult to execute and the least common of the two, attack an application by flooding the memory space reserved for a program.

Background: Encapsulating Security Payload (ESP) is a protocol in the Internet Protocol Security (IPsec) family that encrypts and authenticates data packets sent between computers via a virtual private network (VPN). VPNs can work securely because of the emphasis and layers on which ESP functions.

When one protocol’s packets or frames are encapsulated within another protocol, it increases the overall frame size. Encapsulation adds a protocol header, so any packets that are created at 1500 bytes and are then encapsulated will exceed MTU the network can handle. For example (see below):

IPsec encryption performed by the DMVPN adds 73 bytes for ESP-AES-256 and ESP-SHA-HMAC overhead (overhead depends on transport or tunnel mode and the encryption/authentication algorithm and HMAC)

Ref: What is Iotlb?
IOMMUs include an input/output translation lookaside buffer (IOTLB) to speed-up address resolution, but still every IOTLB cache-miss causes a substantial increase in DMA latency and performance degradation of DMA-intensive workloads.

Vulnerability details: In the Linux kernel before 5.16.15, there is a buffer overflow in ESP transformation in net/ipv4/esp4.c and net/ipv6/esp6.c via a large message. In some configurations, local users can gain privileges by overwriting kernel heap objects.

A heap buffer overflow flaw was found in IPsec ESP transformation code in net/ipv4/esp4[.]c and net/ipv6/esp6[.]c. This flaw allows a local attacker with a normal user privilege to overwrite kernel heap objects and may cause a local privilege escalation threat.

Remedy: The bugfix is ready for download at git.kernel.org. The best possible mitigation is suggested to be upgrading to the latest version.

Observation: Presumably firewall and VPN vendors will suffer in this matter.