Official Updated 12/12/2025 02:20 PM
Preface: Given Google’s efforts to make its hardware compatible with PyTorch, why is the NVIDIA Resiliency Extension (NVRx) still needed? The answer lies in the complementarity of these technologies, the realities of large-scale distributed systems, and the practical necessity to ensure training efficiency and fault tolerance, especially when using NVIDIA GPUs, which remain the mainstream platform for AI development. While Google is working to make its Tensor Processing Units (TPUs) more compatible with PyTorch to provide an alternative, the NVIDIA Resiliency Extension aims to address the specific and pressing challenges encountered when performing large-scale distributed PyTorch training on NVIDIA’s widely used hardware, such as hardware on cloud platforms like Google Cloud.
Background: The open-source machine learning framework PyTorch was originally developed by researchers at Facebook AI Research (now Meta AI). It was first publicly released in September 2016. PyTorch-based workloads are artificial intelligence (AI) and machine learning (ML) tasks and applications that are developed, trained, and run using the PyTorch open-source deep learning framework.
The NVIDIA Resiliency Extension (NVRx) integrates multiple resiliency-focused solutions for PyTorch-based workloads. Users can modularly integrate NVRx capabilities into their own infrastructure to maximize AI training productivity at scale. The NVIDIA Resiliency Extension (NVRx) is a specific Python package that integrates several solutions to minimize downtime and maximize the effective training time for PyTorch-based workloads running on NVIDIA infrastructure. NVRx is a Python package that framework developers and users can modularly integrate into their own infrastructure. It is used in major NVIDIA frameworks like the NVIDIA NeMo Framework for building large language models, providing the underlying machinery for their resilient training capabilities.
NVRx provides utilities that run the actual checkpoint saving routines in the background. It employs mechanisms, often leveraging torch.multiprocessing, to fork a separate, temporary process dedicated to handling the I/O operations after the data has been quickly staged from GPU memory to CPU buffers.
Vulnerability details: CVE-2025-33235 – NVIDIA Resiliency Extension for Linux contains a vulnerability in the checkpointing core, where an attacker may cause a race condition. A successful exploit of this vulnerability may lead to information disclosure, data tampering, denial of service, or escalation of privileges.
Official announcement: Please refer to the link for details – https://nvidia.custhelp.com/app/answers/detail/a_id/5746