PaSh: Light-touch Data-Parallel Shell Processing

A system for parallelizing POSIX shell scripts. Hosted by the Linux Foundation.

Quick Jump: Running PaSh | Installation | Testing | Repo Structure | Community & More | Citing

Installation via pip (Recommended)

pip install binpash-pash
pash -c "echo hi"

Running PaSh

To parallelize, say, ./evaluation/intro/hello-world.sh with parallelization degree of 2× run:

pash -w 2 ./evaluation/intro/hello-world.sh

If the script contains bash specific syntax, add the beta --bash flag to enable support.

Run pash --help to get more information about the available commands. Jump to docs/tutorial for a longer tutorial.

Local testing (development)

To install and run PaSh for local development:

pip install -e .
pash --help
pash -c "echo hello | cat"
./scripts/run_tests.sh

For more details, manual installation, or other platforms see installation instructions.

Running with a local annotations library

To run with a local version of the library, please refer to the documentation local annotations setup and usage

Repo Structure

This repo hosts the core pash development. The structure is as follows:

• src/pash/: Main Python package (installed via pip)
  • preprocessor: Parses shell scripts, expands variables, and identifies dataflow regions for compilation.
  • compiler: Translates shell dataflow regions to IRs and applies parallelization transformations.
  • jit_runtime: Just-in-time runtime that executes compiled regions and manages shell state.
  • runtime: Runtime components — e.g., eager, split, and associated combiners.
• docs: Design documents, tutorials, installation instructions, etc.
• evaluation: Shell pipelines and example scripts used for the evaluation.
• scripts: Scripts related to continuous integration, deployment, and testing.

Community & More

Chat:

• Discord Server (Invite)

Mailing Lists:

• pash-devs: Join this mailing list for discussing all things pash
• pash-commits: Join this mailing list for commit notifications

Development/contributions:

• Contribution guide: docs/contributing
• Continuous Integration Server: ci.binpa.sh

Citing

If you used PaSh, consider citing the following paper:

@inproceedings{pash2021eurosys,
author = {Vasilakis, Nikos and Kallas, Konstantinos and Mamouras, Konstantinos and Benetopoulos, Achilles and Cvetkovi\'{c}, Lazar},
title = {PaSh: Light-Touch Data-Parallel Shell Processing},
year = {2021},
isbn = {9781450383349},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3447786.3456228},
doi = {10.1145/3447786.3456228},
pages = {49–66},
numpages = {18},
keywords = {POSIX, Unix, pipelines, automatic parallelization, source-to-source compiler, shell},
location = {Online Event, United Kingdom},
series = {EuroSys '21}
}