LEARNING HUB
CURRENT TREND INSIGHT
How to clear local system GPU VRAM cache when model crashes Illustration

How to clear local system GPU VRAM cache when model crashes

Written by Nikhil Badjatya
Direct Summary:

When a local model crash leaves VRAM occupied, the cause is almost always a Python/CUDA process that's still alive (or died without releasing its memory allocations) rather than a permanent hardware lock. The fix is layered: try torch.cuda.empty_cache() from a live process first, but when a crashed process has orphaned its allocation, you need to find and kill the actual process holding the memory via nvidia-smi, since empty_cache() only releases memory back to the driver from within the same process that allocated it.

"In God we trust; all others bring data."

— W. Edwards Deming

Key Insights

  • empty_cache() only works within the same process: PyTorch's caching allocator holds freed memory for reuse rather than returning it to the OS immediately — torch.cuda.empty_cache() releases that cached (but unused) memory back to the driver, but it can't touch memory held by a different, already-crashed process.
  • A crashed script doesn't always release its VRAM: if a Python process segfaults or is killed abruptly (rather than exiting cleanly), the CUDA context can remain until the process is fully reaped — nvidia-smi will still show it holding memory.
  • Fragmentation, not just leaks, causes OOM: PyTorch's default caching allocator can fragment VRAM under varying input sizes, so you may see "10GB free" reported while a 2GB allocation still fails — PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is a documented mitigation.

"Clear the VRAM cache" is really two different problems wearing the same name. If your Python process is still running and just accumulated unused cached memory, torch.cuda.empty_cache() handles it. If your process already crashed and nvidia-smi still shows it holding gigabytes of memory, no in-process call can fix that — you have to find the process ID and kill it directly, because the CUDA context it created is what's actually holding the allocation.

Diagnosing and recovering from GPU memory lockup

1. Check what's actually holding memory. Run nvidia-smi and look at the process list at the bottom of the output — it shows every process with an active CUDA context and how much memory each one holds.

2. If the process is alive, free cached memory in-process. Call torch.cuda.empty_cache() and, if you're holding references to old tensors, explicitly del them first — empty_cache() only returns memory PyTorch has already marked as free, not memory still referenced by a live tensor.

3. If the process is dead but memory is still held, kill it directly. Use the PID from nvidia-smi with kill -9 <pid> (Linux) or Task Manager / taskkill /PID <pid> /F (Windows) — this is the actual fix for the "crashed but VRAM still full" case, not a cache-clearing call.

vram_recovery.sh 
# 1. See which processes are actually holding GPU memory
nvidia-smi

# Example output includes a process table like:
#   PID   Type   Process name        GPU Memory Usage
#   14231 C      python3             6142MiB

# 2. If that PID belongs to a crashed/zombie process, kill it directly
kill -9 14231

# 3. In a live Python session, free cached (but unused) memory instead
import torch
del model, batch          # drop references first
torch.cuda.empty_cache()   # then release the cache
Situation Correct Fix Why empty_cache() Alone Fails
Live process, memory fragmented/unused del stale tensors, then torch.cuda.empty_cache() N/A — this is the case it's designed for
Process crashed, VRAM still shown as used Find the PID in nvidia-smi and kill it directly The allocating process no longer exists to call the API in

The underlying lesson generalizes past GPUs: any "just clear the cache" fix only works if the process that owns the resource is still alive to run it. Once a process has crashed or hung, the operating system's process table — not the framework's in-memory API — is the source of truth for what's actually still holding a resource.

Practical Challenge

Deliberately trigger a CUDA OOM (allocate a tensor larger than your free VRAM), then run nvidia-smi to confirm whether the crashed process released its memory automatically or needs a manual kill.

Concept Check

Why can't torch.cuda.empty_cache() fix VRAM still shown as occupied by a process that has already crashed?
Correct! The fix for a crashed process holding VRAM is to identify and kill the actual PID via nvidia-smi, not to call an in-process API that no longer has a process to run in.
Incorrect. Try again! The key distinction is process ownership: a live process can free its own cache, but a crashed one needs to be killed at the OS level via its PID.

Sources & Further Reading

Previous Guide Dashboard Next Guide