Fix use-after-free races in memory pool shrinker and DRM fence destruction

[neoyubi]

Fix use-after-free races in memory pool shrinker and DRM fence destruction

Summary

This patch fixes two related use-after-free race conditions that cause kernel crashes under memory pressure:

1. Memory pool shrinker race: kswapd can invoke shrinker callbacks while nv_mem_pool_destroy() is freeing pool resources
2. DRM fence/GEM destruction race: Kernel drm_exec/shrinker infrastructure can access dma_resv while fence contexts are being destroyed

Both issues stem from the same root cause: cleanup callbacks not being stopped before the resources they access are released.

---

Issue 1: Memory Pool Shrinker Race

Problem

The shrinker is unregistered after freeing the pool's page lists:

void nv_mem_pool_destroy(nv_page_pool_t *mem_pool)
{
    // Free dirty pages, stop scrubber, free clean pages
    // SHRINKER STILL REGISTERED - CALLBACKS CAN FIRE!

    nv_mem_pool_shrinker_free(mem_pool);  // Too late
}

Race Scenario

nv_mem_pool_destroy()                kswapd
─────────────────────                ──────
Free dirty_list
                                     shrink_slab() calls shrinker callback
                                     Callback accesses freed lists
                                     USE-AFTER-FREE
shrinker_free()

Fix

1. Move nv_mem_pool_shrinker_free() to the start of destruction
2. Add synchronize_rcu() after unregistration to ensure no callbacks are in-flight (kernel iterates shrinkers under RCU)
3. NULL the shrinker pointer immediately after unregistration

---

Issue 2: DRM Fence Context Destruction Race

Problem

When a GEM object with an associated fence context is destroyed, the current code:

1. Calls drm_gem_object_release() (releases dma_resv)
2. Then stops callbacks and signals fences

This allows the kernel's drm_exec/shrinker infrastructure to access dma_resv while fences are still active.

Race Scenario

nv_drm_gem_free()                    Kernel drm_exec/shrinker
─────────────────                    ────────────────────────
drm_gem_object_release()
  → dma_resv released
                                     Iterates dma_resv for eviction
                                     Accesses fence context
                                     USE-AFTER-FREE
Stop callbacks, signal fences
Free fence context

Fix

Introduce two-phase destruction for fence contexts:

1. prepare_release/prepare_destroy: Stop callbacks, timers, and signal all pending fences before drm_gem_object_release()
2. free/destroy: Release NVKMS resources and free memory after the GEM object is fully released

This ensures fences are detached from dma_resv before the kernel can no longer safely access them.

---

Changes

nv-vm.c

• Move shrinker unregistration to start of nv_mem_pool_destroy()
• Add synchronize_rcu() after shrinker_free()/unregister_shrinker()
• NULL shrinker pointer after unregistration

nvidia-drm-gem.h/c

• Add prepare_release callback to nv_drm_gem_object_funcs
• Call prepare_release before drm_gem_object_release() in nv_drm_gem_free()

nvidia-drm-fence.c

• Add prepare_destroy callback to nv_drm_fence_context_ops
• Split __nv_drm_prime_fence_context_destroy() into prepare/destroy phases
• Split __nv_drm_semsurf_fence_ctx_destroy() into prepare/destroy phases
• Implement __nv_drm_fence_context_gem_prepare_release() to call prepare phase

---

Testing

• Hardware: NVIDIA RTX 5090 (Blackwell architecture)
• Driver: nvidia-open 590.48.01
• Kernel: 6.18.5 (Arch Linux)
• Before patch: Random system freezes every few hours to days, crashes in kswapd path through nvidia shrinker/fence callbacks
• After patch: Stable under sustained memory pressure and GPU workloads

---

Impact

These bugs affect all users of nvidia-open kernel modules under memory pressure. Symptoms include:

• Random system freezes requiring hard reboot
• Kernel panic in shrink_slab() or drm_exec paths
• Page fault at nvidia module addresses during memory reclaim

The fixes follow established kernel conventions: unregister/stop callbacks before freeing the resources they access.

---

References

• Linux kernel shrinker API: include/linux/shrinker.h
• DRM GEM object lifecycle: drivers/gpu/drm/drm_gem.c
• Affected files:
  • kernel-open/nvidia/nv-vm.c
  • kernel-open/nvidia-drm/nvidia-drm-gem.c
  • kernel-open/nvidia-drm/nvidia-drm-gem.h
  • kernel-open/nvidia-drm/nvidia-drm-fence.c

[CLAassistant]

All committers have signed the CLA.

[Binary-Eater]

Reading through this code, I believe the refactors to nvidia-drm's nv_drm_fence_context_ops and nv_drm_gem_object_funcs does not help much. Would you mind doing a couple of things?

1. Could you share the callstack/kernel panic trace where you believe this change is helping? Preferably a full collection using sudo nvidia-bug-report.sh if possible, but I realize under a kernel panic that may not be possible.
2. Could remove all the changes to nvidia-drm but keep your other changes and retest? I am curious if you see the same calltraces or not.