BLAS: bad multi-threaded performance for herk and hemm routines with OpenBLAS in the 2023a toolchain

[smoors]

as noted by @casparvl here: #268 (comment)

• the herk and hemm routines have much worse multi-threaded performance than the other routines.
• their performance is the same as the single-threaded performance, while for the other routines performance scales with number of threads
• this bad perf seems to happen only for 2023a, 2022b and 2024a are fine.

as can be seen in the output here: #268 (comment) ,
the issue seems to be with larger matrices. initially the perf increases with increasing matrix size, but then suddenly from 1400 x 1400 the perf breaks down.

would be good to check the OpenBLAS repo to see if this is a known regression

[casparvl]

Also tested with 2023b (OpenBLAS 0.3.24), it does not contain the issue:

P: sgemm: 692.05 GFLOPS (r:0, l:None, u:None)
P: shemm: 896.35 GFLOPS (r:0, l:None, u:None)
P: sherk: 771.41 GFLOPS (r:0, l:None, u:None)
P: strmm: 932.43 GFLOPS (r:0, l:None, u:None)
P: strsm: 750.71 GFLOPS (r:0, l:None, u:None)
P: dgemm: 430.87 GFLOPS (r:0, l:None, u:None)
P: dhemm: 438.88 GFLOPS (r:0, l:None, u:None)
P: dherk: 390.95 GFLOPS (r:0, l:None, u:None)
P: dtrmm: 437.85 GFLOPS (r:0, l:None, u:None)
P: dtrsm: 334.21 GFLOPS (r:0, l:None, u:None)
P: cgemm: 1189.46 GFLOPS (r:0, l:None, u:None)
P: chemm: 1176.96 GFLOPS (r:0, l:None, u:None)
P: cherk: 941.37 GFLOPS (r:0, l:None, u:None)
P: ctrmm: 1047.02 GFLOPS (r:0, l:None, u:None)
P: ctrsm: 946.88 GFLOPS (r:0, l:None, u:None)
P: zgemm: 593.17 GFLOPS (r:0, l:None, u:None)
P: zhemm: 593.21 GFLOPS (r:0, l:None, u:None)
P: zherk: 467.05 GFLOPS (r:0, l:None, u:None)
P: ztrmm: 549.19 GFLOPS (r:0, l:None, u:None)
P: ztrsm: 490.12 GFLOPS (r:0, l:None, u:None)

[casparvl]

Question is if it's an openblas or GCC issue. We don't see anything related to the release notes in 0.3.24 that would have fixed this https://github.com/OpenMathLib/OpenBLAS/blob/develop/Changelog.txt . There is some mention of working around a tree-vectorize issue (though not specifically for these routines). But maybe it's a GCC-introduced issue? One way to find out is to compile OpenBLAS with the GCC from 2023b. If it still doesn't work, it must be specific to the OpenBLAS issue. If it does fix things, it's relate to the GCC version.