Failures of QuantumESPRESSO on all systems that submit results to the dashboard

[laraPPr]

We should take a look at what is going with these tests but it does not seem stable. Both at UGent and Surf it only seem to pass the tests sporadically in the last few months.

I will take a look at the output and errors of those test at UGent to track down what is going wrong on our systems.

[laraPPr]

It also seems that we stopped testing QE regurlarly on the Magic Castle cluster because it has much less nodes for this test than others.

[laraPPr]

It looks like QE is not being run at the UGent systems. It hits the timewall but it only seems like the EESSI environment was set up. After the setup I'm seeing this in the output.

node4101.gallade.os:rank0.pw.x: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted
node4101.gallade.os:rank0.pw.x: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted
node4102.gallade.os:rank1.pw.x: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted
node4102.gallade.os:rank1.pw.x: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted

[laraPPr]

At UGent e were missing the lmod hook, https://www.eessi.io/docs/known_issues/eessi-2023.06/#failed-to-modify-ud-qp-to-init-on-mlx5_0-operation-not-permitted. We added it yesterday and the newest runs on our tier1 already seem a lot more stable. On our Tier2 I still see a lot of failures so investigating that further.

Don't know yet what is going wrong at surf.

[laraPPr]

Should have checked if the hook was actually their because it was still missing on Tier2.