Tracing torch.cuda.empty_cache() on an RTX 4090 - Where Do the 53 MB Go?

**Análise Técnica: Tracing torch.cuda.empty_cache() no RTX 4090**

Olá, comunidade de webmastersmz.com! Estou aqui para discutir um tópico interessante relacionado à tecnologia de processamento de dados em GPUs. O artigo "Tracing torch.cuda.empty_cache() on an RTX 4090 - Where Do the 53 MB Go?" nos leva a uma jornada para entender como funciona a limpeza do cache de memória da GPU NVIDIA RTX 4090.

O método `torch.cuda.empty_cache()` é usado para liberar a memória cache da GPU, o que pode ser útil para evitar problemas de memória insuficiente durante a execução de modelos de aprendizado de máquina. No entanto, a pergunta que o artigo busca responder é: onde vão os 53 MB de memória que são liberados quando chamamos essa função?

A análise técnica realizada no artigo mostra que a memória liberada não é necessariamente "perdida" ou "destruída", mas sim realocada para outras partes do sistema. Isso ocorre porque a GPU tem várias camadas de cache e buffers que armazenam dados temporários durante a execução de operações de processamento.

Os principais pontos a serem destacados são:

* A GPU NVIDIA RTX 4090 tem uma arquitetura complexa, com várias camadas de cache e buffers que trabalham juntas para otimizar o desempenho.
* O método `torch.cuda.empty_cache()` libera a memória cache da GPU, mas não necessariamente elimina os dados armazenados nessa memória.
* A memória liberada é realocada para outras partes do sistema, como a memória principal do sistema ou outros buffers da GPU.

Essa análise técnica é importante para entender como funciona a gestão de memória em GPUs e como otimizar o desempenho de modelos de aprendizado de máquina que utilizam esses dispositivos.

**Convidando à Discussão**

Agora, gostaria de convidar a comunidade de webmastersmz.com a discutir esse tópico e compartilhar suas experiências com a gestão de memória em GPUs. Qual é a sua experiência com o uso de GPUs em projetos de aprendizado de máquina? Você já teve problemas de memória insuficiente e como os resolveu?

**Serviços de Alojamento de Alta Performance**

Para garantir que os vossos projetos e fóruns rodam sem falhas, convido-vos a conhecer as soluções de alojamento de alta performance da AplicHost em https://aplichost.com. Com opções de alojamento personalizadas e suporte técnico especializado, a AplicHost pode ajudar a garantir que os vossos projetos sejam executados de forma eficiente e segura. Visite o site da AplicHost para saber mais sobre as soluções de alojamento de alta performance e como elas podem beneficiar o seu negócio.

Tracing torch.cuda.empty_cache() on an RTX 4090 - Where Do the 53 MB Go?



Tópico: Tracing torch.cuda.empty_cache() on an RTX 4090 - Where Do the 53 MB Go?
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------


TL;DR


After del tensor; torch.cuda.empty_cache(), PyTorch's caching allocator still holds 53.7 MB that it won't release. We traced the CUDA Runtime and Driver APIs with eBPF uprobes to see exactly what happens at the kernel level during the free path. The trace showed cudaFree calls hitting p99 = 1.9ms (4.6x their median) because the process keeps getting descheduled mid-free. The allocator isn't broken - the OS is interrupting it.



The Issue


pytorch/pytorch#173382 - a user calls torch.cuda.empty_cache() after deleting tensors, but GPU memory stays allocated. The caching allocator's empty_cache() only releases blocks it has marked as free, but the user sees a persistent gap between "allocated" and "reserved" memory. We traced what happens when torch cuda empty cache runs on an RTX 4090 and measured exactly how much GPU memory it reclaims.

The docs say it releases "unoccupied cached memory." But how do you tell which blocks are occupied, which are free, and what's holding them?



Reproducing It


We wrote a small script that loads Qwen2.5-0.5B-Instruct, runs 3 inference rounds, and logs CUDA memory at each step. RTX 4090, PyTorch 2.10, NVIDIA driver 580.

# After each inference round:
del output_ids
del input_ids
torch.cuda.empty_cache()

The output:

[after model load              ] allocated=   950.2 MB  reserved=   992.0 MB  gap=    41.8 MB
[round 1: after generate       ] allocated=   958.3 MB  reserved=  1020.0 MB  gap=    61.7 MB
[round 1: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB
[round 2: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB
[round 3: after del+empty_cache] allocated=   958.3 MB  reserved=  1012.0 MB  gap=    53.7 MB
[after del model+empty_cache   ] allocated=     8.1 MB  reserved=    20.0 MB  gap=    11.9 MB
[after gc.collect+empty_cache  ] allocated=     8.1 MB  reserved=    20.0 MB  gap=    11.9 MB

The 53.7 MB gap stays constant across all 3 rounds. empty_cache() reclaims some memory (reserved drops from 1020 to 1012 MB) but never closes the gap. Even after deleting the model and running gc.collect(), 11.9 MB remains unreachable.

This is exactly what the issue reporter described. But the numbers don't explain why.



What nvidia-smi Shows


Nothing useful. nvidia-smi reports total GPU memory usage but can't see inside PyTorch's caching allocator. torch.cuda.memory_snapshot() gives block-level info, but mapping blocks back to specific cudaMalloc calls or figuring out what's holding a reference is painful.

We wanted to see the actual cudaMalloc and cudaFree calls happening at the driver level.



Tracing with eBPF


We attached eBPF uprobes to libcudart.so and libcuda.so to trace every CUDA memory operation, kernel launch, and synchronization call. The trace also captures Linux scheduler events (context switches, wakeups) so we can see when the process gets preempted.

# Start trace (captures CUDA Runtime + Driver + host scheduler events)
sudo ./bin/ingero trace --duration 90s &

# Run the workload while tracing
python3 cuda_empty_cache_leak.py

The trace captured 2.7 MB of data across the full inference cycle.



Watch the Full Investigation


MiniMax M2.7 autonomously investigating the PyTorch empty_cache trace data via the MCP interface. Watch full interactive recording on asciinema



What the Trace Showed


Five causal chains, all pointing to the same root cause:

Operation
P50
P99
Slowdown
What It Means

cudaMemcpyAsync
9 us
887 us
98.6x
Memory copies stall when thread gets preempted

cudaFree
413 us
1.9 ms
4.6x
Free operations slow down mid-execution

cudaLaunchKernel
8 us
25 us
3.2x
Kernel launches delayed

cudaStreamSync
3 us
22 us
6.9x
Sync waits inflated

The trace recorded 288 context switches during the workload. Every time the Python process was descheduled by the Linux scheduler, whatever CUDA operation was in progress got delayed.

The key finding: cudaFree calls hit p99 = 1.9ms (4.6x their median of 413us). When empty_cache() iterates over free blocks and calls cudaFree for each one, the process can get preempted mid-iteration. The allocator isn't stuck - it's being interrupted.



The Actual Problem


It's two things stacked:

• PyTorch's caching allocator holds blocks for reuse by design. The 53.7 MB gap is blocks that are allocated at the CUDA level but not currently backing any Python tensor. The allocator keeps them because reallocating GPU memory is expensive. empty_cache() releases these, but only the ones the allocator has marked as truly free.

• The host CPU is interfering with the free path. When empty_cache() does run, system services (journald, atopacct, resolved) on the same machine compete for CPU time. The cudaFree calls take 4.6x longer at p99 because the thread gets descheduled mid-operation.

The first part is by design. The second part makes it worse on shared machines - cloud VMs, containers, or any environment with noisy neighbors.



What We Learned


The allocator is doing what it's supposed to. The gap between "allocated" and "reserved" is the caching allocator's working set - blocks it holds for fast reallocation. empty_cache() can only release blocks that have no active references, and the 53.7 MB consists of blocks the allocator decided to keep.

The 11.9 MB that persists even after deleting the model and running gc.collect is likely CUDA context overhead - driver-internal allocations that PyTorch doesn't control.

If you are hitting this in production, the fix is not a force=True parameter on empty_cache. It is understanding that the caching allocator is a feature, not a bug. If you genuinely need that memory back (e.g., to load a second model), delete all references, call gc.collect(), then empty_cache(). If the gap persists, those blocks have active references somewhere - possibly in autograd state, CUDA graphs, or internal PyTorch buffers.



Try It Yourself


Clone the repo and connect any MCP-compatible AI:

# 1. Build
git clone https://github.com/ingero-io/ingero.git
cd ingero && make build

# 2. Create the MCP config (points to this post's investigation DB)
cat > /tmp/ingero-mcp.json << 'EOF'
{
"mcpServers": {
"ingero": {
"command": "./bin/ingero",
"args": ["mcp", "--db", "investigations/pytorch-173382-empty-cache.db"]
}
}
}
EOF

# 3. Install ollmcp (MCP client for Ollama)
pip install ollmcp

# 4. Investigate with a local model
ollmcp -m qwen3:32b -j /tmp/ingero-mcp.json

Type /investigate to start the guided workflow. The repro script is at tests/workloads/pathological/cuda_empty_cache_leak.py.

GitHub (give us a star!): github.com/ingero-io/ingero. No NVIDIA SDK, no code changes, production-safe by design.

If you are seeing unexpected behavior from PyTorch memory management, we would love to take a look. Drop an issue on GitHub and we will dive into it together.

Ingero is free & open source software licensed under Apache 2.0 (user-space) + GPL-2.0/BSD-3 (eBPF kernel-space). One binary, zero dependencies, <2% overhead.



Related reading


• Tracing a 13x PyTorch slowdown from hidden NumPy synchronization

• Debugging PyTorch GPU out-of-memory errors

• 124x slower PyTorch DataLoader traced at kernel level