fix: PSv2 follow-up fixes from integration tests

[mihow]

Summary

The PSv2 seems to work sometimes, but some events cause no tasks to be returned, NATS to timeout or Django to become unresponsive.

Here are some findings from my session 2 out of 3. More to come!

---

Fixes discovered during PSv2 integration testing where stale ADC workers overwhelmed Django by flooding /tasks with concurrent NATS reserve requests.

• NATS get_connection() now uses connect_timeout=5, allow_reconnect=False to prevent leaked reconnection loops from blocking Django's async event loop
• /tasks endpoint returns empty for terminal-status jobs instead of attempting NATS reserve
• IncompleteJobFilter excludes jobs by top-level status (not just progress JSON stages), so workers don't pick up manually-failed jobs
• Integration test script kills stale ADC workers before starting and uses AMI_NUM_WORKERS=0

Context

During integration testing, a stale ADC worker from a previous test run consumed all 20 NATS messages before the new worker could fetch them. The stale worker spawned multiple DataLoader subprocesses, generating 147 concurrent /tasks requests against Django's single uvicorn worker thread. Combined with NATS connections that defaulted to allow_reconnect=True, this blocked the entire event loop and made Django unresponsive.

TODO & TO-TRY

• Add timeouts to JetStream _ensure_stream() / _ensure_consumer() operations
• Wrap async_to_sync() NATS calls with asyncio.wait_for() timeout
• NATS connection pooling (see PSv2: Use connection pooling and retries for NATS #1130 for prior attempt)
• Add --workers 4 to uvicorn dev config (optional? currently 1 is required for debugger)
• /tasks multi-pipeline support
• ADC: use trailing slashes to avoid 301 redirects
• Set dispatch_mode on job init, not run()
• Update how the status of processing services are checked & reported #1122 processing service online status
• feat: PSv2 - Have visibility into which workers are subscribed to a job #1112 worker tracking in logs

Test plan

• Run scripts/psv2_integration_test.sh 20 with clean state (no stale workers)
• Verify Django stays responsive during worker task fetching
• Verify terminal-status jobs return empty tasks without hitting NATS
• Verify incomplete_only=1 excludes FAILURE/SUCCESS/REVOKED jobs

Full findings: docs/claude/planning/nats-flooding-prevention.md

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	17d4c6b
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/699416cc04a9c9000829012d

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	17d4c6b
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/699416cc6107f90008b6fcef

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/nats-connection-safety

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}