Assistance for codecheckers

[nuest]

I would like to collect and discuss resources and ideas here about possible means for assisting codecheckers in the efforts. I would still say this is not about automation, though ideally these tools are running "in the cloud" somehow, alternatively as CLI tools or possibly as part of a bot.

Both codecheckers and authors can easily point them to a repository or a PDF to get advise/starting points for a check or for improving their work.

[nuest]

Papercheck (https://scienceverse.github.io/papercheck) is an R package for checking research papers, e.g., it checks if retracted or replicated papers are cited, if pvalues are reported correctly and other checks (mostly statistics).

It could be interesting to add a module for basic codechecking using the same methods (ML, LLM, pattern matching):

• is code mentioned
• is data mentioned (input, output)
• are figures based on code
• does the code/data it have a licence
• does the code have a readme
• is it in a code repository and also deposited with a DOI
• is it properly cited
• is there a link to an online notebook
• ...

Or to think how to expand this for engineering sciences?

(via @DanielaGawehns 🙏)

[Richie2154]

Statcheck can help with checking p-values.

[nuest]

That's right! We are careful about things like these though, because we are deliberately not checking the correctness of methods - but this is a quick thing to do, so we should consider putting it on the list.

[nuest]

Reproducibility as a Service (RaaS) by @jwons could be useful for codecheckers to use, but also for authors to run pre-checks before they submit?

https://doi.org/10.1002/spe.3202

(from codecheckers/discussion#30)

[nuest]

See code published as part of this study: https://www.nature.com/articles/s41597-022-01143-6

From the same team, a more powerful automated checking platform: https://arxiv.org/abs/2109.10387

Also for R: https://github.com/ropensci-review-tools/pkgcheck

[nuest]

Inspired by

O’Brien, G. Threats to scientific software from over-reliance on AI code assistants. Nat Comput Sci 5, 701–703 (2025). https://doi.org/10.1038/s43588-025-00845-2

Ideas for prompts to use during a CODECHECK (e.g., with Copilot or Claude Code, working on the local codebase)

• Does the parametrisation and configuration presented in the article match the settings in the code?
• How do the workflow steps described in the manuscript differ from the steps implemented in the code? Is anything missing in either?
• If there is none, create a reproducible computational environment with pinned versions
• What are errors or potentially risks in the code?
• Re-do the ... statistical tests from the workflow in a Python/R (another language) and compare the results with the original
• Identify similar workflows in online code repositories that apply similar methods and ...
• ...