Simplify and Clarify Job State Model for Improved Consistency and Documentation

[pesap]

Summary

This issue proposes simplifying and clarifying the job state model to reduce ambiguity, remove overlapping meanings, and improve both documentation and implementation.

Proposed changes

• Reduce overlapping terminal states (canceled / terminated)
  Both represent jobs that are intentionally stopped before completion. Unless there is a strong requirement to distinguish user-initiated from system-initiated termination, these can likely be collapsed into a single terminal state (for example, canceled).

• Reduce overlapping dependency states (blocked / uninitialized)
  Both represent jobs whose dependencies are not yet satisfied. The distinction between “created but not evaluated” and “evaluated but still waiting” may not be necessary. We could merge it into a single state that simply means “created but not runnable due to dependencies.”

• Remove disabled as a runtime state
  The disabled state does not describe an execution phase, but rather a configuration/metadata flag. A disabled job means “do not schedule this job,” which is better modeled as a property (for example, enabled = false) instead of a runtime state in the lifecycle.

• Clarify the distinction between ready and pending in the documentation
  The intended behavior seems to be:

  • ready: job has satisfied all dependencies and is waiting in the queue to be claimed by a runner.
  • pending: job has been claimed and is waiting for resources to start running.
    Example wording for pending: “Job was claimed and is waiting for resources to run.”

[daniel-thom]

In my mind there is an important distinction between canceled and terminated that follows the convention of SIGTERM on UNIX operating systems.

Canceled:

• User performs the cancel operation. Jobs never run.
• A job fails. A job that was dependent on its output is automatically canceled. It will not be run until the user fixes the problem.

Terminated:

• Not yet implemented. Was present in the old torc.
• A compute node is reaching the wall time timeout and sends SIGTERM to the torc worker.
• The torc worker sends SIGTERM to all running jobs.
• The running jobs gracefully shut down in such a way that a re-run can resume from the checkpoint.

Terminated was always a theoretical case. No user implemented the required functionality. Is it worth the extra status? It still might have value for new users.

Blocked vs Uninitialized:
The initialize step will fail if required input files are not present. We do automatically initialize for the user in run/submit. I do think there is some valuable information here.

Disabled:
I see your point. It was present in the old torc but is not currently implemented.

Ready vs pending:
I have debated changing pending to claimed. This status only exists in the small window between the server granting the job and the worker starting it. The worker is not waiting on anything. We could theoretically go straight to running, but I thought it might be useful to designate this temporary condition in case something fails.