[Automated ingestion] Don't verify signatures in constructor, only do it for certain states

[Neves-P]

On the monitoring channel, we started getting a lot of [FIRING:1] (HostUnusualDiskReadRate monitor.eessi.io rug-nl-s0.eessi.science multixscale-stratum0 warning) warnings.

When I logged to the stratum 0 server, I could see with htop that every few minutes processes using ssh-keygen would show up and do about 450M/s reads and then stop again. This was happening for tarballs of staging PRs that had already been merged, but whose tarballs were still in /srv/tmp/. Looking closer, I could then see things like:

eessi     349637  347378  0 15:29 ?        00:00:00 /bin/bash /opt/eessi/sign_verify_file_ssh.sh --verify --allowed-signers-file /opt/eessi/allowed_signers --file /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/skylake_avx512/accel/nvidia/cc80/17552616730/eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-accel-nvidia-cc80-17552616730.tar.gz --signature-file /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/skylake_avx512/accel/nvidia/cc80/17552616730/eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-accel-nvidia-cc80-17552616730.tar.gz.sig
eessi     349674  349637 92 15:30 ?        00:00:05 ssh-keygen -Y verify -f /opt/eessi/allowed_signers -n eessi-bot-deucalion -I EESSI -s /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/skylake_avx512/accel/nvidia/cc80/17552616730/eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-accel-nvidia-cc80-17552616730.tar.gz.sig
pedro     349676  348836  0 15:30 pts/0    00:00:00 grep --color=auto sign

I have no clue why this is being triggered, but I did edit crontab -e the day before to test the new ingestion workflow. I also ran the ingestion script manually a couple of times, and while I don't remember stopping it halfway, maybe I did.

To try and work around the issue, I'll delete these tarballs, since they're in the bucket and have been ingested anyway

[Neves-P]

sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/software/linux/aarch64/a64fx/17551628270/
sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/sapphirerapids/accel/nvidia/cc70/17552597530/
sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/skylake_avx512/17543944960/
sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/software/linux/x86_64/intel/skylake_avx512/accel/nvidia/cc70/17552588970

Removed these tarballs from the /srv/tmp directory. Let's see how it goes. It still doesn't address any possible cause

[Neves-P]

This didn't solve the issue... I will now do:

sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/
sudo rm -rf /srv/tmp/tarballs-bundles/2025.06/

We don't have any staging PRs open, so it should be okay.

[Neves-P]

I deleted every tarball and immediately I have some of them again

[pedro@multixscale-stratum0 grace]$ ls -la
total 24
drwxr-xr-x. 6 eessi eessi 4096 Aug 20 15:47 .
drwxr-xr-x. 3 eessi eessi 4096 Aug 20 15:47 ..
drwxr-xr-x. 2 eessi eessi 4096 Aug 20 15:47 17544272540
drwxr-xr-x. 3 eessi eessi 4096 Aug 20 15:47 1755617085
drwxr-xr-x. 2 eessi eessi 4096 Aug 20 15:47 17556324460
drwxr-xr-x. 3 eessi eessi 4096 Aug 20 15:47 accel

Before checking for files I had done:

[pedro@multixscale-stratum0 tarballs-bundles]$ #sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/
[pedro@multixscale-stratum0 tarballs-bundles]$ sudo rm -rf /srv/tmp/tarballs-bundles/2023.06/
[pedro@multixscale-stratum0 tarballs-bundles]$ sudo rm -rf /srv/tmp/tarballs-bundles/2025.06/
[pedro@multixscale-stratum0 tarballs-bundles]$ sudo rm -rf /srv/tmp/tarballs-bundles/*.tar.gz*
[pedro@multixscale-stratum0 tarballs-bundles]$ ls -l

so there shouldn't be anything there.

This is what we have:

[pedro@multixscale-stratum0 linux]$ pwd
/srv/tmp/tarballs-bundles/2023.06/software/linux
[pedro@multixscale-stratum0 linux]$ tree -L 4
.
├── aarch64
│   └── nvidia
│       └── grace
│           ├── 17544272540
│           ├── 1755617085
│           ├── 17556324460
│           └── accel
└── x86_64
    └── amd
        └── zen2
            ├── 17543945010
            ├── 1755615859
            ├── 1755616593
            ├── 1755616959
            ├── 17556318260
            └── accel

16 directories, 0 files

Note that there are no open staging PRs or deployments ongoing, and these tarballs are part of already merged PRs such as https://github.com/EESSI/staging_bundles/pull/11