Buffers created using GBM are not usable when imported into non-Nvidia drivers

[hoshinolina]

NVIDIA Open GPU Kernel Modules Version

580.119.02

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Fedora Linux 43 (KDE Plasma Desktop Edition)

Kernel Release

Linux homura 6.18.6-200.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Jan 18 18:57:00 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA RTX A400 (UUID: GPU-aeefd27a-9d0b-2ff2-aeb2-5ff1e89d9dfc)

Describe the bug

When a (linear) buffer is created using GBM on the Nvidia driver, it can be imported successfully into a non-Nvidia card. However, actually attempting to sample from the texture on the non-nvidia side fails during command submission (presumably while preparing/locking the BOs used for the submission).

AMDGPU hangs retrying infinitely, with this error looping in dmesg:

amdgpu: [drm] *ERROR* Not enough memory for command submission!

Intel eventually complains in stderr:

intel: the execbuf ioctl keeps returning ENOMEM

Other combinations work:

• GBM allocate on Nvidia, import & Vulkan blit on Nvidia, import & GL sample on Nvidia
• GBM allocate on AMD, import & Vulkan blit on AMD, import & GL sample on Nvidia
• GBM allocate on AMD, import & Vulkan blit on AMD, import & GL sample on AMD
• GBM allocate on AMD, import & Vulkan blit on Nvidia, import & GL sample on AMD
• GBM allocate on Nouveau, import & Vulkan blit on Nouveau/NVK, import & GL sample on AMD

So it seems it's specifically something about Nvidia-side GBM allocations that makes them not usable by other drivers.

I first noticed this when trying to use libfunnel running on an Nvidia card to share buffers with OBS via PipeWire. libfunnel works by doing a GBM allocation on the local (source) GPU, importing it into Vulkan by fd (on the same GPU), blitting to it, then sending the GBM dma-buf FD to the destination via PipeWire (where it will be imported, possibly into a different GPU).

This has been reproduced by other people on other GPU combinations/systems.

To Reproduce

This is a reduced test case: https://gist.github.com/hoshinolina/9d0731cba9fad23562dda90106178cf8

It expects two DRM device names, one where the texture will be allocated, and one where it will be imported and then drawn using EGL (using external textures so it works on nvidia too):

gcc -o gbm-export-test gbm-export-test.c -lGL -lEGL -lgbm
./gbm-export-test [SRCDEV] [DSTDEV]

• ./gbm-export-test /dev/dri/renderD(nvidia) /dev/dri/renderD(nvidia): works
• ./gbm-export-test /dev/dri/renderD(amd) /dev/dri/renderD(amd): works
• ./gbm-export-test /dev/dri/renderD(amd) /dev/dri/renderD(nvidia): works
• ./gbm-export-test /dev/dri/renderD(nouveau) /dev/dri/renderD(amd): works
• ./gbm-export-test /dev/dri/renderD(nvidia) /dev/dri/renderD(amd): hangs with dmesg errors

(Note that, for simplicity, this test doesn't render anything to the texture on the source GPU, it just allocates it and leaves it uninitialized. This is not relevant, the case where I first discovered this was doing a proper Vulkan blit on the source side.)

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

[hoshinolina]

I enabled DRM debugging and that shows that the ENOMEM comes from nvidia:

[2094080.696041] [drm:__nv_drm_gem_nvkms_memory_prime_get_sg_table [nvidia_drm]] [GPU ID 0x00000a00] Cannot create sg_table for NvKmsKapiMemory 0x00000000faa659e0
[2094080.696050] amdgpu 0000:06:00.0: amdgpu: [drm] *ERROR* Not enough memory for command submission!

It seems the buffers are allocated from vidmem, and the driver does not know how to migrate them to sysmem when they need to be used by another GPU.

[hoshinolina]

Specifically, it seems the implementation of dma_buf_ops.attach is incorrect. Per the docs:

Exporters which support buffer objects in special locations like VRAM or device-specific carveout areas should check whether the buffer could be move to system memory (or directly accessed by the provided device), and otherwise need to fail the attach operation.

It seems the nv_dma_buf_attach() implementation is not failing, but the driver does not know how to move the BO to sysmem. Therefore, buffer imports succeed, but then it breaks badly when the buffers actually need to be used, since the sgtable is queried at that point and that fails since the buffer has no sysmem pages.

[hoshinolina]

Searching for __nv_drm_gem_nvkms_memory_prime_get_sg_table and Cannot create sg_table for NvKmsKapiMemory yields lots of hits, mostly related to Chromium, for over two years, including this forum thread: https://forums.developer.nvidia.com/t/550-54-14-cannot-create-sg-table-for-nvkmskapimemory-spammed-when-launching-chrome-on-wayland/284775

It seems this was tracked internally as bug 4545633. Nvidia employee "amrits" in that thread reported "So far, it does not look like the hang happens in our drivers.". Assuming it's the same issue as here, this is an incorrect diagnosis. The "hang" occurs because the non-nvidia driver is infinitely retrying an operation, which calls into a callback into the Nvidia driver, which fails when it shouldn't. While the retry loop is not in the Nvidia driver, the cause of the hang is a bug in the Nvidia driver.