Skip to content

fix: PSv2 - Tasks are queued, worker sees job but no tasks #1123

@mihow

Description

@mihow

Summary

When an async job is started, it appears that the tasks get queued, and the worker sees the job, but the worker sees zero tasks to complete.

Detailed Description

In Antenna, I start a job and it appears that all tasks are queued successfully:

[2026-02-08 02:17:04] INFO Initialized task state tracking for 15 images
[2026-02-08 02:17:04] INFO Queuing 15 images to NATS stream for job '2225'
[2026-02-08 02:17:04] INFO Shuffling images
[2026-02-08 02:17:04] INFO Found 15 out of 31 images to process

In the worker, the job is seen in the API, but there are zero tasks to complete.

2026-02-08 07:19:28 [info     ] Checking for jobs for pipeline quebec_vermont_moths_2023
2026-02-08 07:19:28 [info     ] Processing job 2225 with pipeline quebec_vermont_moths_2023
2026-02-08 07:19:28 [info     ] Worker 0/1 starting iteration for job 2225
2026-02-08 07:19:29 [info     ] Worker 0: No more tasks for job 2225
2026-02-08 07:19:29 [info     ] Worker 0: Iterator finished
2026-02-08 07:19:29 [info     ] Done, detections: 0. Detecting time: 0.0, classification time: 0.0, dl time: 0.0, save time: 0.0

How to Reproduce the Bug

It would be helpful to expose the NATS dashboard for the live environment.

Environment

https://antenna.insectai.org/projects/18/jobs/2225

Steps to reproduce the behavior:

  1. Start a job, wait for the tasks to be queues
  2. View the logs for the worker service on one of the GPU worker VMs

Screenshots

Image Image

Additional context

Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions