vulkan: Warptile tuning for Intel Xe2/Xe3 #18178
Conversation
github-actions
bot
added
Vulkan
| if ((device->vendor_id == VK_VENDOR_ID_INTEL) && (device->driver_id == vk::DriverId::eIntelProprietaryWindows)) { | ||
| if (device->coopmat_support && device->architecture == INTEL_XE2) { | ||
| // Xe2/Xe3 with coopmat enabled - warptile performance tuning | ||
| m_warptile = { 512, 128, 128, 16, subgroup_size_8, 32, 2, tm_m, tn_m, tk_m, subgroup_size_8 }; |
There was a problem hiding this comment.
I wonder if this should actually be the large tile size?
Also, a quick google search suggests Xe2 has 64KB of register file per core, which with 512 invocations is only 32 registers each which seems very low. But I've never worked on this hardware so I'm just speculating.
There was a problem hiding this comment.
Hi @jeffbolznv Sure I can look at re-enabling large warptile size for Intel here and then moving the warptile config from m_ to l_. I'll also check perf again after the change.
Are you doing (64 * 1024) / 512 invocations is 128 bytes per invocation and the assumption is 4 byte width register? (to get 32 registers per invocation?)
There was a problem hiding this comment.
Yes, that's the calculation I did.
There was a problem hiding this comment.
Thanks Jeff, for Xe architecture each register in GRF was 32 Byte wide. But I need to look into the register situation a bit deeper
|
CC @mmerecki in case this makes sense to also enable for Linux. |
ggml/src/ggml-vulkan/ggml-vulkan.cpp
Outdated
| m_align = 64; | ||
| s_align = 32; | ||
|
|
||
| if ((device->vendor_id == VK_VENDOR_ID_INTEL) && (device->driver_id == vk::DriverId::eIntelProprietaryWindows)) { |
There was a problem hiding this comment.
Ah it's good to see more tunes show up 😃. Please move your tune to line 2845 so that they're all placed together.
There was a problem hiding this comment.
Hi @netrunnereve thanks! I can do that but the wg_denoms change also needs to overwrite the default and be 'paired' with this tuning config to pass unit tests. Do you want me to make two separate 'if' statements with the same conditions?
There was a problem hiding this comment.
I would just move this section above line 2841.
l_mmq_wg_denoms = l_wg_denoms = {128, 128, 1 };
m_mmq_wg_denoms = m_wg_denoms = { 64, 64, 1 };
s_mmq_wg_denoms = s_wg_denoms = { 32, 32, 1 };
l_align = 128;
m_align = 64;
s_align = 32;
|
Oh and please run a before and after |
Thanks! I checked on Arc B580, everything saw good improvement (except for type_a=bf16 test which was same perf) 
My IGPU had same perf on all quants as it doesn't support coopmat |
3 participants
Summary
In this PR we modify the m_warptile configurations to increase prompt processing performance for Intel Xe2+ GPUs that support coopmat extension
Changes
Accuracy Check
Basic testing w/ llama-cli across models that show perf jump (+ trying different prompt sizes) - model output looks correct / reasonable.
Unit tests Check
Checked on system w/ Arc B580 + Intel Integrated Graphics. All unit tests pass.
Performance Results
Command ran is
llama-bench.exe -m ..\<model> -p 512 -r 5 -n 128The eval token gen results don't change and weren't expected to, only prompt processing :) Below numbers show prompt processing in tok/s for Arc B580 and Lunar Lake Series 2 IGPU.
PR Status
Ready for Review