ntb_perf v2.0 not working with a 2 port NTB link on kernel 5.4

[jborz27]

I'm seeing ntb_hw_switchtec driver not working with ntb_perf 2.0 on kernel 5.4.

According to ntb_perf documentation writing the port number to the /sys/kernel/debug/ntb_perf/[bdf]/run should start the performance test.

Here are excerpts from both the ntb_perf info parameter and dmesg logs from both hosts.

Host 1
cat /sys/kernel/debug/ntb_perf/0000:01:00.1/info
Performance measuring tool info:

Local port 0, Global index 0
Test status: idle
Port 0 (0), Global index 0:
Link status: up
Out buffer addr 0xffffb34e84500000
Out buffer size 0x0000000000080000
Out buffer xlat 0x0000000000000000[p]
In buffer addr: unallocated

dmesg
[118700.910614] ntb_perf 0000:01:00.1: Global port index 0
[118700.910619] ntb_perf 0000:01:00.1: Message service unsupported
[118700.910620] ntb_perf 0000:01:00.1: Scratchpad service initialized
[118700.910628] ntb_perf 0000:01:00.1: DB bits unmasked 0x1
[118700.910632] switchtec switchtec0: enabling link
[118700.912932] switchtec switchtec0: message: 0 00000003
[118700.913178] switchtec switchtec0: doorbell
[118700.913495] ntb_perf 0000:01:00.1: DB vec 0 mask 0xfffffff bits 0x1
[118700.915226] ntb_perf 0000:01:00.1: CMD exec: 0
[118700.915236] switchtec switchtec0: ntb link up
[118700.915469] ntb_perf 0000:01:00.1: CMD send: 0 0x80000

Host 2
cat /sys/kernel/debug/ntb_perf/0000:03:00.1/info
Performance measuring tool info:

Local port 0, Global index 0
Test status: idle
Port 0 (0), Global index 0:
Link status: up
Out buffer addr 0xffffb272c1400000
Out buffer size 0x0000000000080000
Out buffer xlat 0x0000000000000000[p]
In buffer addr 0xffff9b0def300000
In buffer size 0x0000000000080000
In buffer xlat 0x00000003ef300000[p]

dmesg
[198323.821290] switchtec switchtec0: ntb link up
[198323.821528] ntb_perf 0000:03:00.1: CMD send: 0 0x80000
[198323.821532] ntb_perf 0000:03:00.1: DB ring peer 0x1
[198323.823662] switchtec switchtec0: message: 0 00000003
[198323.823879] switchtec switchtec0: doorbell
[198323.824210] ntb_perf 0000:03:00.1: DB vec 0 mask 0xfffffff bits 0x1
[198323.824214] ntb_perf 0000:03:00.1: CMD recv: 0 0x80000
[198323.824216] ntb_perf 0000:03:00.1: CMD exec: 1
[198323.824867] switchtec switchtec0: MW 0: part 0 addr 0x00000003ef300000 size 0x0000000000080000
[198323.944368] ntb_perf 0000:03:00.1: CMD exec: 2
[198323.944373] ntb_perf 0000:03:00.1: CMD send: 2 0x3ef300000

[lsgunth]

There were a bunch of fixes after v5.4 for ntb_perf. As I recall it was broken for a while in that era. You probably need to use a newer kernel.

[jborz27]

Any recommendation for which kernel to use?

[lsgunth]

Try the latest.

[jborz27]

Got ntb_perf to work on kernel 5.13: The DMA results are at default so may be able to yield even better throughput.

I did need to unload/load ntb_perf twice on one of the servers to get the in/out buffers to be setup on both ends of the NTB link.

Non-DMA Performance (Gen3 x8)
cat /sys/kernel/debug/ntb_perf/0000:03:00.1/run
Peer 0 test statistics:
0: copied 1073741824 bytes in 207997 usecs, 5162 MBytes/s (65% theoretical max)

DMA Performance (Gen3 x8)
cat /sys/kernel/debug/ntb_perf/0000:09:00.1/run
Peer 0 test statistics:
0: copied 1073741824 bytes in 156451 usecs, 6863 MBytes/s (87% theoretical max)