iMarina-load

Scripts to obtain A3 data, transform it into iMarina load format, and upload it to iMarina server using SFTP
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
   • Run program
   • Run in host
   • Run in Docker
4. Testing
   • Prerequisites
   • Execute tests
   • In host
   • Other useful testing commands
   • In Docker 🐳
   • In CI/CD automated testing (GitHub actions)
5. Roadmap (issues)
6. Contributing
7. Top contributors
8. License
9. Contact

(back to top)

Built with

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

(back to top)

Getting Started

Follow these steps to set up the project locally.

Prerequisites

Install Python version 3.12.3 or above, git and other essentials for building the project.

In Ubuntu is:

 sudo apt install python3.12-venv gcc build-essential git -y

Installation

Clone repository

git clone https://github.com/ICIQ-DMP/iMarina-load.git

Initialize venv

cd iMarina
make install

Obtaining translations

First you will need to obtain the spreadsheets with the translations. By default, they are read from the input/ folder, but you can change the location of these expected files with the following arguments:

• --imarina-input /path/to/iMarina.xlsx
• --a3-input /path/to/A3.xlsx
• --countries-dict /path/to/countries.xlsx
• --jobs-dict /app/input/Job_Descriptions.xlsx

You can either download them manually from Sharepoint, or you can use the Dockerized OneDrive service to sync files from Sharepoint to your host computer automatically in the background.

To do so you should do the following:

cd services/onedrive
bash run.sh
# The program will display a link and ask you to authenticate and paste the answered URL into the terminal

Institutional Strengthening/_Projects/iMarina_load_automation/input from Sharepoint into services/onedrive/data. Add or change the necessary arguments to read from this new source, instead of input/, so that data consumed by the program is always updated.

You can leave OneDrive running so that the files are always in sync.

There are two configuration options active in services/onedrive/conf/config to make OneDrive delete things that have been deleted in Sharepoint cleanup_local_files = "true" and to only do downloads, not uploads (one-way sync) download_only = "true". You may remove these two options to change the behaviour from one-way sync to two-way sync.

(back to top)

Usage

Run program

Run in host

To start the program execute this command:

./venv/bin/python src/main.py 

Run in Docker

Use the provided Dockerfile and compose.yml to build and run the iMarina-load service in a containerized environment.

Dockerfile Builds a lightweight Python 3.12 Alpine image that installs dependencies and runs the main script with predefined input file paths.

compose.yml Defines a service that builds and runs the iMarina-load container, mounts input/output folders, and securely injects FTP credentials as secrets for automated data processing.

First, you will need to create a .env file at the root of the project with the UID and GID of the user on your host that looks like this:

UID=1015
GID=1015

We used 1015 as example, but you can create a .env with the correct values with:

cat <<EOF > .env
UID=${UID:-$(id -u)}
GID=${GID:-$(id -g)}
EOF

In any case, you need to pass the variables UID and GID to docker compose to make it work.

To build the Docker image and run it you can use:

  docker compose up --build

Other useful commands:

Build Docker image

  sudo docker build . -t aleixmt/imarina-load --progress=plain

Access the container shell

  docker compose run --rm app sh

(back to top)

Testing

Prerequisites

Install the requirements-dev.txt to install the dependencies for the tests.

make dev

Execute tests

In host

We have to be at the root of the project, otherwise we will get an error

cd ~/Desktop/iMarina-load

After that, we can use this to run all tests at the same time:

pytest -v

Other useful testing commands

• Display prints or logs during tests:

pytest -s -v

• Stop at the first fail in the test:

pytest -x

• Run specific tests for example (by name):

pytest -k "name"

In Docker 🐳

Run the following command to build the test image and execute the tests against it:

docker compose -f compose.yml -f compose.test.yml up --build 

In CI/CD automated testing (GitHub actions)

Our project uses GitHub Actions to run the tests against every pushed commit to the master branch.

The workflow installs dependencies, runs all pytest tests, and builds the Docker image only if all tests pass. The workflow is defined in .github/workflows/docker.yml

(back to top)

Roadmap (issues)

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are welcome and what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you’d like to report a bug, request a feature, or propose an improvement, please follow these steps:

Create an Issue

Create a new Issue in here.

• Title: A short, descriptive summary of the issue.
• Description: Provide as much context as possible.
• For bugs: steps to reproduce, expected vs. actual behavior, environment (OS, Python version, etc.).
• For features: explain the motivation and the expected outcome.
• Screenshots or logs (if applicable).

The maintainers will review it and may ask for further clarification.

Create a Pull Request

Fork the repository, implement the changes that you want on your fork and create a Pull Request in here.

The maintainers will try to integrate it into the master branch.

(back to top)

Top contributors:

License

Distributed under the GNU GPL v3. See LICENSE for more information.

(back to top)

Contact

• Mario Piqué - mpique@iciq.es
• Aleix Mariné - amarine@iciq.es

(back to top)