[Bug]: Enabling B-frames in FFMPEG hangs up with HQVBR

[pbaykalov]

Describe the bug
Enabling B-frames in FFMPEG hangs up with HQVBR on frame 1.
ffmpeg -hwaccel d3d11va -i in.mp4 -c:v h264_amf -usage transcoding -rc hqvbr -b 8M -preanalysis true -bf 3 out.mp4 - hangs up
ffmpeg -hwaccel d3d11va -i in.mp4 -c:v h264_amf -usage transcoding -rc cbr -b 8M -preanalysis true -bf 3 out.mp4 - works fine
ffmpeg -hwaccel d3d11va -i in.mp4 -c:v h264_amf -usage transcoding -rc hqvbr -b 8M -preanalysis true out.mp4 - works fine

To Reproduce
Steps to reproduce the behavior:

1. ffmpeg -hwaccel d3d11va -i in.mp4 -c:v h264_amf -usage transcoding -rc cbr -b 8M -preanalysis true -bf 3 out.mp4
2. Encoding hangs up indefinitely.

Setup (please complete the following information):

• OS: W11 24H2
• Driver Version 25.6.1
• GPU 7900XTX
• Encoder: ffmpeg 8.0.1

Debug Log (please upload or paste):
N/A

Expected behavior
Can't see why B-frames should not work with HQVBR specifically.

Screenshots
N/A

Additional context
I've set up HQVBR in OBS 32 successfully with B-frames enabled.

[MikhailAMD]

• Could you please share detailed FFmpeg log?
• Could you please try AMF decoder? -hwaccel amf -hwaccel_output_format amf
• Please try to increase pool size: -hwaccel d3d11va -extra_hw_frames 30

[pbaykalov]

Thank you!
I should have said that hwaccel has zero impact on the problem, even dropping it entirely does not fix the issue.

ffmpeg  -i .\out.mp4 -c:v h264_amf -usage transcoding -rc hqvbr -b 8M -preanalysis true -bf 3 temp.mp4
ffmpeg version 8.0.1-full_build-www.gyan.dev Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 15.2.0 (Rev8, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-shared --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-lcms2 --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-libdvdnav --enable-libdvdread --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libopenjpeg --enable-libquirc --enable-libuavs3d --enable-libxevd --enable-libzvbi --enable-liboapv --enable-libqrencode --enable-librav1e --enable-libsvtav1 --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxeve --enable-libxvid --enable-libaom --enable-libjxl --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-openal --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-liblc3 --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint --enable-whisper
  libavutil      60.  8.100 / 60.  8.100
  libavcodec     62. 11.100 / 62. 11.100
  libavformat    62.  3.100 / 62.  3.100
  libavdevice    62.  1.100 / 62.  1.100
  libavfilter    11.  4.100 / 11.  4.100
  libswscale      9.  1.100 /  9.  1.100
  libswresample   6.  1.100 /  6.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf62.3.100
  Duration: 00:04:43.57, start: 0.000000, bitrate: 6181 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6002 kb/s, 59.99 fps, 60 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 164 kb/s, start 0.018000 (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Please use -b:a or -b:v, -b is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_amf))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
2026-01-27 21:01:01.6748092 [V:onnxruntime:Default, env.cc:885 onnxruntime::WindowsEnv::InitializeCpuInfo] Found total 8 core(s) from windows system:
2026-01-27 21:01:01.6770067 [V:onnxruntime:Default, env.cc:886 onnxruntime::WindowsEnv::InitializeCpuInfo]
core 1 consist of logical processors: 1 2
core 2 consist of logical processors: 3 4
core 3 consist of logical processors: 5 6
core 4 consist of logical processors: 7 8
core 5 consist of logical processors: 9 10
core 6 consist of logical processors: 11 12
core 7 consist of logical processors: 13 14
core 8 consist of logical processors: 15 16
Output #0, mp4, to 'temp.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf62.3.100
  Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 8000 kb/s, 60 fps, 15360 tbn (default)
    Metadata:
      encoder         : Lavc62.11.100 h264_amf
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      encoder         : Lavc62.11.100 aac
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
frame=    1 fps=0.1 q=-0.0 size=       0KiB time=-00:00:00.01 bitrate=N/A speed=N/A elapsed=0:00:11.33

Maybe it will be also useful to mention that there is zero codec utilization.

[MikhailAMD]

-loglevel verbose
-extra_hw_frames 30
-hwaccel amf -hwaccel_output_format amf