Preface: DeepSpeed MII, an open-source Python library developed by Microsoft, aims to make powerful model inference accessible, emphasizing high throughput, low latency, and cost efficiency. TensorRT LLM, an open-source framework from NVIDIA, is designed for optimizing and deploying large language models on NVIDIA GPUs.
Microsoft’s DeepSpeed suite (including DeepSpeed-MII) and NVIDIA’s TensorRT-LLM each use their own proprietary kernel optimizations, memory management systems, and compilation workflows.
Background: During the early development lifecycle of TensorRT-LLM (v0.x through early v1.x), the framework relied heavily on PyTorch routines to read incoming checkpoint binaries primarily due to an aggressive development timeline focused on rapid time-to-market and deep integration with the existing Hugging Face/PyTorch ecosystem.Rather than building custom, bare-metal file deserializers from scratch, NVIDIA engineers utilized PyTorch as a bridge to quickly establish functional features.
The early reliance on standard PyTorch routines (like torch.load()) to read checkpoint binaries is directly related to why an initialized RestrictedUnpickler did not mandate a strict Register Allowlist restriction (Safe Globals).The technical breakdown of why this occurred during the v0.x through early v1.x lifecycle includes:
The Open-World “Arbitrary Tensor” ProblemTo implement a strict RestrictedUnpickler with an active Register Allowlist, a developer must know every single class, module, and data type that will ever appear in the input file.
The Reality: In early TensorRT-LLM releases, scripts had to support checkpoints coming from Hugging Face, deep-speed saves, raw PyTorch, Megatron-LM, and dozens of community-forked models.
The Conflict: If NVIDIA engineers had mandated a strict Safe Globals allowlist, any checkpoint containing custom third-party extensions, custom metadata classes, or non-standard tensor storages (like specialized quantization state structures) would have instantly triggered a _pickle.UnpicklingError. Mandating it would have severely hindered user adoption.
Vulnerability details: CVE-2026-24233 NVIDIA TensorRT-LLM for Linux contains a vulnerability in the restricted unpickler used for model weight deserialization, where a local, unauthenticated attacker could cause deserialization of untrusted data. A successful exploit of this vulnerability might lead to code execution, escalation of privileges, data tampering, and information disclosure.
Official announcement: Please refer to the link fore details – https://nvidia.custhelp.com/app/answers/detail/a_id/5840