AMD (Genoa) architecture not recognized

[hsane-dev]

When trying to run on m7a.8xl or 32 core Genoa, the tool (binary or build from scratch) results in "architecture not recognized", although the specific architecture files can be found.

Please find the paste from the log file:

time=2025-01-03T23:15:02.439Z level=INFO msg="Starting up" app=perfspect version=3.1.0_2024-12-28_789175b2 PID=2591 arguments="./perfspect metrics"
time=2025-01-03T23:15:02.758Z level=INFO msg="Using Linux perf" target=hsane-tools path=/tmp/perfspect.tmp.2933821012/perf
time=2025-01-03T23:15:02.792Z level=ERROR msg="error running script on target" script="/tmp/perfspect.tmp.2933821012/perf stat -a -e cpu/event=0x2a,umask=0x01,offcore_rsp=0x104004477,name='OCR.READS_TO_CORE.LOCAL_DRAM'/ sleep 1" stdout="" stderr="event syntax error: '..umask=0x01,offcore_rsp=0x104004477,name=OCR.READS_TO_C..'\n \___ unknown term 'offcore_rsp' for pmu 'cpu'\n\nvalid terms: event,edge,inv,umask,cmask,config,config1,config2,name,period,percore,metric-id\n\nInitial error:\nevent syntax error: '..umask=0x01,offcore_rsp=0x104004477,name=OCR.READS_TO_C..'\n \___ unknown term 'offcore_rsp' for pmu 'cpu'\n\nvalid terms: event,edge,inv,umask,cmask,config,config1,config2,name,period,percore,metric-id\nRun 'perf list' for a list of valid events\n\n Usage: perf stat [] []\n\n -e, --event event selector. use 'perf list' to list available events\n" exitcode=129 error="exit status 129"
time=2025-01-03T23:15:03.791Z level=WARN msg="Uncore devices not supported"
time=2025-01-03T23:15:03.803Z level=WARN msg="ref-cycles not supported" output=""
time=2025-01-03T23:15:03.803Z level=WARN msg="Fixed-counter TMA events not supported" output="\n Performance counter stats for 'system wide':\n\n 0 TOPDOWN.SLOTS \n 0 PERF_METRICS.BAD_SPECULATION \n\n 1.002642280 seconds time elapsed\n\n"
time=2025-01-03T23:15:03.805Z level=ERROR msg="error loading metadata" error="failed to determine if fixed-counter 'cpu-cycles' is supported: unsupported uarch: Genoa" target=hsane-tools
time=2025-01-03T23:15:03.805Z level=ERROR msg="error loading metadata" error="failed to determine if fixed-counter 'instructions' is supported: unsupported uarch: Genoa" target=hsane-tools
time=2025-01-03T23:15:03.805Z level=ERROR msg="error loading metadata" error="failed to determine if 'OCR' is supported: failed to determine if ocr is supported: event syntax error: '..umask=0x01,offcore_rsp=0x104004477,name=OCR.READS_TO_C..'\n \___ unknown term 'offcore_rsp' for pmu 'cpu'\n\nvalid terms: event,edge,inv,umask,cmask,config,config1,config2,name,period,percore,metric-id\n\nInitial error:\nevent syntax error: '..umask=0x01,offcore_rsp=0x104004477,name=OCR.READS_TO_C..'\n \___ unknown term 'offcore_rsp' for pmu 'cpu'\n\nvalid terms: event,edge,inv,umask,cmask,config,config1,config2,name,period,percore,metric-id\nRun 'perf list' for a list of valid events\n\n Usage: perf stat [] []\n\n -e, --event event selector. use 'perf list' to list available events\n, 129, exit status 129" target=hsane-tools
time=2025-01-03T23:15:03.805Z level=ERROR msg="failed to prepare metrics" target=hsane-tools error="target not supported, see log for details"
time=2025-01-03T23:15:03.805Z level=ERROR msg="no targets had metrics prepared"

[harp-intel]

Please test the build found at the bottom of this page on your Genoa systems: https://github.com/intel/PerfSpect/actions/runs/12613246868

[hsane-dev]

This build still fails with the following error:
$ sudo ./perfspect metrics live
hsane-tools ⣯ Error: failed to put perf events into groups: unrecognized event format: "Workload failed: No such file or directory"

If 'live' is not used, then it collects but most metrics result in "NaN" values. "perf stat" itself works on the system and reports CPI, which fails with perfspect. I am assuming that events are not being recognized and hence not grouped - resulting in error forming the perf-stat command for collection.

[harp-intel]

Hi @hsane-dev. Please see the comment on the PR, here: #139 (comment)

Does this output match what you are seeing on m7a.8xl? Please run with the --debug flag to capture detailed logs. Thanks.

I haven't tried 'live' mode on m7a.8xl...looks like a different problem. I'll check into it.

[harp-intel]

@hsane-dev With regard to basic metrics resulting in NaN, We are getting "<not counted>" and "<not supported>" values from perf stat on many events, including cpu-cycles and instructions. The event (grouping) definition files for Genoa were provided to us by AMD engineers for bare metal (not EC2) systems. I suspect that they are not compatible with EC2 m7a.

[harp-intel]

@hsane-dev Some groups are successfully reading cpu-cycles and instructions, but PerfSpect's algorithm for selecting the groups to use for a particular metric does not take into consideration the event values (within the group) during selection. So, if those values are "<not counted>", then the metric value will be NaN. I believe this can be improved. I'll look into it.

[harp-intel]

Improved with small change to event group selection for metrics:

More testing required.

[harp-intel]

$ sudo ./perfspect metrics live

I figured this one out... "live" needs to be "--live" otherwise perfspect is interpreting it as a program to run.

[hsane-dev]

Yes, the event grouping is causing the issue. I tested the first group with grouping {} and without. I also removed just one event from the group and then it counts. The right grouping of events and testing needs to be done prior to having perfspect launch it.

sudo perf stat -a -e '{cpu/event=0x120,umask=0x1,name='ls_not_halted_p0_cyc'/,cpu/event=0x120,umask=0x1,name='ls_not_halte
d_p0_cyc:k'/k,cpu-cycles,cpu-cycles:k,instructions,instructions:k}'
^C
Performance counter stats for 'system wide':

 <not counted>      ls_not_halted_p0_cyc                                                  
 <not counted>      ls_not_halted_p0_cyc:k                                                
 <not counted>      cpu-cycles                                                            
 <not counted>      cpu-cycles:k                                                          
 <not counted>      instructions                                                          

instructions:k

   6.482325925 seconds time elapsed

The events in group usually have to be from the same PMU. Try reorganizing the group.

hsane-tools hsane ~ $ sudo perf stat -a -e cpu/event=0x120,umask=0x1,name='ls_not_halted_p0_cyc'/,cpu/event=0x120,umask=0x1,name='ls_not_halted_
p0_cyc:k'/k,cpu-cycles,cpu-cycles:k,instructions,instructions:k
^C
Performance counter stats for 'system wide':

 2,885,970,588      ls_not_halted_p0_cyc                                                    (83.34%)
 2,377,101,381      ls_not_halted_p0_cyc:k                                                  (83.33%)
 3,862,555,548      cpu-cycles                                                              (83.33%)
 3,184,498,387      cpu-cycles:k                                                            (83.34%)
 2,430,429,763      instructions                     #    0.63  insn per cycle              (83.34%)
 1,646,531,809      instructions:k                   #    0.52  insn per cycle              (83.34%)

   4.185820328 seconds time elapsed

[harp-intel]

@hsane-dev I created a new issue here: #143