[Feature Request]: Implement Graceful Shutdown mechanism for Hop Server (especially for Kubernetes/Containerized environments)

[petertsneg]

What would you like to happen?

Currently, hop-server.sh supports a -k flag to stop the server, but it acts more as a kill switch than a graceful shutdown. In containerized environments like Kubernetes, when a Pod is being updated or scaled down, the platform sends a SIGTERM signal to the process.

Without a graceful shutdown mechanism, ongoing pipelines and workflows are interrupted abruptly, which may lead to data inconsistency or incomplete processing.

Proposed Solution / Requirements:

Signal Handling: Allow Hop Server to catch SIGTERM signals and trigger a shutdown sequence.

Wait for Completion: Provide an option/parameter to allow running pipelines/workflows to finish their current tasks within a configurable timeout period before the server process exits.

Reject New Tasks: Once the shutdown sequence is initiated, the server should stop accepting new execution requests.

Use Case: This feature is critical for users running Apache Hop in CI/CD pipelines or cloud-native environments (K8s) to ensure high availability and data integrity during routine deployments and updates.

Issue Priority

Priority: 3

Issue Component

Component: Hop Server

[mattcasters]

• Add a shutdown hook in hop server and hop-server.sh
• Add a command line option
• Add a StopServerServlet servlet
• Add a "shutting down" status for the server
• Refuse new work (adding pipelines, workflows, running, services, ...) on the system while still allowing all status services.
• Wait until all pipelines, workflows and services have completed before shutting down the server.
• Allow configuration of a maximum wait time in seconds until shutdown (0 is immediate and this is the default)

[nadment]

Hi Matt, an old PR exists, but since then there have been many conflicts. I don't know if it would be easier to start from scratch or merge the conflicts if you want to work on it.

#4574

[mattcasters]

Oh yes, now I remember! Doh. Let's get this PR resolved and then I'll implement the shutdown status, work rejection and maximum wait time. Let me know if you need any help merging the code.

[nadment]

If you have the time and desire to revisit this PR and resolve the conflicts, or simply take inspiration from it, please do so.