CVE-2022-23471 containerd CRI stream server: Host memory exhaustion through Terminal resize goroutine leak (7th Dec 2022)

Preface: As of version 1.23, Kubernetes requires runtimes to be CRI compatible. It means that dockershim is now deprecated, and Docker Engine is no longer supported as a runtime. However, Kubernetes can still communicate with Docker via containerd, which can be CRI compliant with a plugin.

History:

  • In October 2014, Docker released the first version.
  • In July 2015, the first version of K8s was released, using Docker as the container runtime.
  • In December 2016, K8s introduced the CRI architecture in v1.5, and separated the logic related to operating Docker to introduced Dockershim.
    The Dockershim is the CRI compliant layer between the Kubelet and the Docker daemon.
  • In May 2022, K8s announces separation from Dockershim
    Remark: Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.

“To not want a Buddhist monk after the ritual is finished”. Reality tells us that in the business world, this situation is not uncommon.

Background: The kubelet is the primary “node agent” that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod.
Cri-containerd is an implementation of CRI for containerd. It operates on the same node as the Kubelet and containerd. Layered between Kubernetes and containerd, cri-containerd handles all CRI service requests from the Kubelet and uses containerd to manage containers and container images.

Vulnerability details:
A bug was found in containerd’s CRI implementation where a user can exhaust memory on the host. In the CRI stream server, a goroutine is launched to handle terminal resize events if a TTY is requested.
If the user’s process fails to launch due to, for example, a faulty command, the goroutine will be stuck waiting to send without a receiver, resulting in a memory leak.
Ref: Kubernetes and crictl can both be configured to use containerd’s CRI implementation and the stream server is used for handling container IO.

Observation: The source file (httpstream.go) in line 127 has description (supportsTerminalResizing returns false because v1ProtocolHandler doesn’t support it). The remedy append the “context” function. Context is used to solve the functions of exit notification and metadata transfer between goroutines.
Another hints provided by software developers state the following. To avoid leaking this routine, exit if the http request finishes. This path would generally be hit if starting the process fails and nothing is started to ingest these resize events.
I speculated that this the additional details of this design weakness.

Resolution: This bug has been fixed in containerd 1.6.12 and 1.5.16. Users should update to these versions to resolve the issue. Users unable to upgrade should ensure that only trusted images and commands are used and that only trusted users have permissions to execute commands in running containers.
Remark: containerd is a container runtime and Docker is a container engine

Official announcement: Please refer to the link for details :
https://github.com/containerd/containerd/security/advisories/GHSA-2qjp-425j-52j9
https://github.com/containerd/containerd/commit/a05d175400b1145e5e6a735a6710579d181e7fb0

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.