Add validation + compatibility tests for DeepLabCut-live by deruyter92 · Pull Request #56 · DeepLabCut/DeepLabCut-live-GUI

deruyter92 · 2026-02-27T08:46:48Z

Motivation:
Since DeepLabCut-live-GUI heavily depends on DeepLabCut-live, but currenly it is implicitly assumed that that package provides the right output. It would be good to add some basic explicit validation of the returned output of DeepLabCut-live to check compatibility of the returned data. Also, including functional compatibility checks in CI for different DeepLabcut-live versions, allows to systematically tets the package compatibility before release.

Changes:

Add PosePacket a lightweight schema validation of the output from DeepLabCut-live
Add tests for package compatibility for different DeepLabCut-live versions
Various CI improvements/robustness fixes

This pull request introduces significant improvements to DLCLive compatibility testing and pose validation in both the codebase and CI infrastructure. The main changes include adding a dedicated compatibility test suite for DLCLive versions, implementing robust pose array validation, and updating the workflow and test environments to support these enhancements.

DLCLive compatibility testing:

Added a new test suite tests/compat/test_dlclive_package_compat.py that verifies DLCLive package importability, constructor signatures, required methods, and performs a minimal inference smoke test. This ensures that the GUI remains compatible with supported DLCLive versions.
Introduced a stub for torch in tests/compat/conftest.py to allow compatibility tests to run even if torch is not installed, improving test robustness.

Pose validation and contract enforcement:

Implemented a new validate_pose_array function in dlclivegui/services/dlc_processor.py to enforce shape and dtype constraints on pose outputs, preventing downstream errors from invalid data.
Updated the processor to use the new pose validation and packet structure, ensuring consistent handling and emission of pose results. [1] [2]
Added unit tests for pose array validation in tests/services/test_pose_contract.py, covering both valid and invalid cases.

Continuous integration and test environment updates:

Added a new dlclive-compat job to .github/workflows/testing-ci.yml to run compatibility tests against multiple DLCLive versions (PyPI and GitHub main), and updated tox configuration to include corresponding environments. [1] [2] [3]
Updated test markers in pyproject.toml to include dlclive_compat and excluded these tests from the default coverage run. [1] [2]

GUI camera validation timer improvements:

Refactored camera validation in dlclivegui/gui/main_window.py to use a cancellable QTimer attribute and ensure it is stopped in closeEvent, reducing test/CI errors related to GUI teardown timing. [1] [2]

These changes improve reliability, maintainability, and future-proofing of the DLCLive integration and pose processing pipeline.

C-Achard

Great work, thanks !

dlclivegui/services/dlc_processor.py

tests/services/test_pose_contract.py

dlclivegui/services/dlc_processor.py

C-Achard

@deruyter92 If you'd like I can merge #55 right now, so then we can see how the tests go in CI ?

deruyter92 · 2026-02-27T12:17:20Z

Yes great thanks. Sorry I see that I didn't install the pre-commit

C-Achard · 2026-02-27T12:41:42Z

Sorry, in the above I meant #55. It is now merged, happy to take care of conflicts/CI if you're busy !

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

Remove the earlier pytest commands block and update the tox 'commands' to run pytest with the marker excluding both 'hardware' and 'dlclive_compat'. Adjust coverage invocation to use the installed package path (--cov={envsitepackagesdir}/dlclivegui) and emit per-env XML coverage files (.coverage.{envname}.xml). This aligns tox behavior with the GitHub Actions job and removes the prior posargs-based command duplication.

.github/workflows/testing-ci.yml

Update the GitHub Actions testing matrix to run on Python 3.12 instead of 3.11. This moves CI to test against the newer Python runtime while keeping existing matrix include entries unchanged.

deruyter92 · 2026-02-28T10:14:35Z

I see that some tests are failing still. I am happy to pick this up later. Importantly this shouldn't hold you back from releasing the bèta-test version. These validation measures can always be added later.

Add a PoseBackends Enum and switch PoseSource.backend from a string to that enum for stronger typing and clarity. Update default PosePacket/PoseSource instances and the validate_pose_array signature to use PoseBackends.DLC_LIVE. Import Enum/auto and remove an unused sentinel variable. These are small refactors to improve type-safety and consistency around backend identification.

Relax and correct DLCLive compatibility checks: only consider keyword/positional params when inspecting __init__, drop the assertion that __init__ must accept **kwargs, and only require 'frame' for init_inference/get_pose (frame_time is passed as a kwarg to processors). Also update FakeDLCLive.get_pose to return an array of shape (2, 3) to match the expected pose output shape.

Set the test step shell to `bash -eo pipefail` and redirect `tox` stderr into `tee` (changed `tox -q` to `tox -q 2>&1 | tee tox-output.log`). This ensures pipeline failures are detected (pipefail) and that tox's stderr is recorded in `tox-output.log` for improved debugging in the CI workflow.

Replace ad-hoc DLCLive installs in the GitHub Actions job with tox-based testenvs. The workflow fixes Python to 3.12, installs required Qt/OpenGL runtime libs on Ubuntu, installs tox and tox-gh-actions, and runs tox -e matrix.tox_env. tox.ini was extended with dlclive-pypi and dlclive-github testenvs (pypi pinned and GitHub main respectively) to run the compatibility pytest, and the new envs were added to env_list to allow local and CI execution.

C-Achard · 2026-03-02T09:06:33Z

@deruyter92 A few additions :

Added a PoseBackend enum to simplify future work with alternate backends
Fixed an older issue/mistake I made in the CI that would result in the CI job passing despite job failure due to piping commands
Routed the compat tests through tox to reduce install steps and keep env consistent.
Let me know what you think !

Add tests/compat/conftest.py to inject a stub torch module into sys.modules when torch is not installed. This prevents ImportError during DLCLive compatibility tests so the API can be validated without requiring torch to be installed. This is a pragmatic workaround and includes a note to remove or replace it once imports are properly guarded in the package.

C-Achard · 2026-03-02T09:20:53Z

I also had to monkeypatch torch in the compat tests due to unguarded imports in DLCLive, see DeepLabCut/DeepLabCut-live#168

Replace QTimer.singleShot with a cancellable QTimer instance (self._camera_validation_timer) that is single-shot, connected to _validate_configured_cameras, and started with a 100ms delay. The closeEvent now stops the timer if it's still active to prevent validation from firing while the window is closing, avoiding modal QMessageBox races/crashes during tests/CI teardown.

C-Achard

Approving again, will wait for your feedback next week @deruyter92 before merging as this is agreed not to be beta-critical; thanks again for all the valuable additions !

Copilot

Pull request overview

This PR adds explicit pose-output contract validation for DeepLabCut-live results and introduces a CI-compatible test suite to verify DeepLabCut-live API/package compatibility across supported versions.

Changes:

Added PosePacket + validate_pose_array() to validate DLCLive pose outputs and emit a structured packet alongside pose data.
Introduced a dlclive_compat pytest marker and new compatibility tests, plus tox environments to run them against PyPI and GitHub-main DLCLive.
Improved CI robustness and GUI teardown reliability (camera validation timer cancellation on close).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`dlclivegui/services/dlc_processor.py`	Adds pose schema/packet and pose array validation integrated into the processing pipeline.
`tests/services/test_pose_contract.py`	Adds unit coverage for pose-array validation (shape/dtype error cases).
`tests/conftest.py`	Updates FakeDLCLive pose shape to match the new (.., 3) pose contract.
`tests/compat/test_dlclive_package_compat.py`	Adds DLCLive import/signature checks and an opt-in smoke test.
`tests/compat/conftest.py`	Adds a torch stub to let compatibility tests run without torch.
`pyproject.toml`	Adds `dlclive_compat` marker definition.
`tox.ini`	Adds dlclive compatibility tox envs and excludes compat tests from default coverage run.
`dlclivegui/gui/main_window.py`	Replaces a one-shot timer with a cancellable timer and stops it in `closeEvent`.
`.github/workflows/testing-ci.yml`	Adds DLCLive compatibility job and improves test step robustness/logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-11T11:14:02Z

tests/compat/test_dlclive_package_compat.py

+def _get_signature_params(callable_obj) -> tuple[set[str], bool]:
+    """
+    Return allowed keyword names for callable, allowing for **kwargs.
+
+    Example:
+    >>> params, accepts_var_kw = _get_signature_params(lambda x, y, **kwargs: None, {"x", "y"})
+    >>> params == {"x", "y"}
+    True
+    >>> accepts_var_kw
+    True
+    """
+    sig = inspect.signature(callable_obj)
+    params = sig.parameters
+    accepts_var_kw = any(p.kind == inspect.Parameter.VAR_KEYWORD for p in params.values())
+    return params, accepts_var_kw


_get_signature_params is annotated as returning tuple[set[str], bool], but it actually returns inspect.Signature.parameters (a mapping of name -> Parameter). The docstring example also calls _get_signature_params(..., {"x", "y"}), but the function only accepts one argument. Please align the annotation/docstring with the actual return value (or change the function to return a set[str] if that’s what you want).

Copilot · 2026-03-11T11:14:02Z

tests/compat/test_dlclive_package_compat.py

+    params, _ = _get_signature_params(DLCLive.__init__)
+    params = {
+        name
+        for name, p in params.items()
+        if p.kind in (inspect.Parameter.KEYWORD_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)
+    }
+    missing = {name for name in expected if name not in params}


The helper returns whether the callable accepts **kwargs (accepts_var_kw), but test_dlclive_constructor_accepts_gui_expected_kwargs ignores it. This means an upstream DLCLive.__init__(**kwargs) implementation would fail this test even though the GUI would still work. Consider treating the expected kwargs as satisfied when accepts_var_kw is true (or explicitly document that **kwargs-only constructors are unsupported).

Suggested change

params, _ = _get_signature_params(DLCLive.__init__)

params = {

name

for name, p in params.items()

if p.kind in (inspect.Parameter.KEYWORD_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)

}

missing = {name for name in expected if name not in params}

params, accepts_var_kw = _get_signature_params(DLCLive.__init__)

params = {

name

for name, p in params.items()

if p.kind in (inspect.Parameter.KEYWORD_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)

}

missing = {name for name in expected if name not in params}

if missing and accepts_var_kw:

# DLCLive.__init__ accepts **kwargs, so GUI-passed kwargs are still supported.

missing = set()

Copilot · 2026-03-11T11:14:02Z

dlclivegui/services/dlc_processor.py

+def validate_pose_array(pose: Any, *, source_backend: PoseBackends = PoseBackends.DLC_LIVE) -> np.ndarray:
+    """
+    Validate pose output shape and dtype.
+
+    Accepted runner output shapes:
+    - (K, 3): single-animal
+    - (N, K, 3): multi-animal
+    """
+    try:
+        arr = np.asarray(pose)
+    except Exception as exc:
+        raise ValueError(
+            f"{source_backend} returned an invalid pose output format: could not convert to array ({exc})"


validate_pose_array types source_backend as PoseBackends, but the new compat smoke test passes a string (e.g. "DLCLive.get_pose"). To avoid type-checking inconsistencies within the repo, either widen the parameter type (e.g. str | PoseBackends) or change the test to pass a PoseBackends value and format a readable label separately.

Copilot · 2026-03-11T11:14:03Z

tests/compat/conftest.py

+
+# Stub out torch imports to avoid ImportError when torch is not installed in DLCLive package.
+# This allows testing of DLCLive API compatibility without requiring torch.
+# Ideally imports should be guarded in the package itself, but this is a pragmatic solution for now.
+# IMPORTANT NOTE: This should ideally be removed and replaced whenever possible.
+if "torch" not in sys.modules:


The torch stub logic checks only "torch" not in sys.modules, which will still be true even if torch is installed but not yet imported; this would shadow the real torch with an empty stub and can cause misleading compatibility results. Prefer try: import torch / except ImportError: ... (or importlib.util.find_spec) so the stub is only used when torch truly isn’t available.

Suggested change

# Stub out torch imports to avoid ImportError when torch is not installed in DLCLive package.

# This allows testing of DLCLive API compatibility without requiring torch.

# Ideally imports should be guarded in the package itself, but this is a pragmatic solution for now.

# IMPORTANT NOTE: This should ideally be removed and replaced whenever possible.

if "torch" not in sys.modules:

import importlib.util

# Stub out torch imports to avoid ImportError when torch is not installed in DLCLive package.

# This allows testing of DLCLive API compatibility without requiring torch.

# Ideally imports should be guarded in the package itself, but this is a pragmatic solution for now.

# IMPORTANT NOTE: This should ideally be removed and replaced whenever possible.

if importlib.util.find_spec("torch") is None:

Copilot · 2026-03-11T11:14:03Z

tox.ini

+# Run locally : tox -e dlclive-pypi
+[testenv:dlclive-pypi]
+description = DLCLive compatibility tests against specific PyPi release
+deps =
+    deeplabcut-live==1.1
+commands =
+    pytest -m dlclive_compat tests/compat/test_dlclive_package_compat.py -q


The compat smoke test is opt-in via DLCLIVE_TEST_MODEL_PATH / DLCLIVE_TEST_MODEL_TYPE, but tox will not pass these env vars through by default. If you intend to run the smoke test under tox (locally or in CI), add these variables to passenv for the dlclive-* envs (or at least document that the smoke test can’t be enabled under tox).

Copilot · 2026-03-11T11:14:03Z

dlclivegui/services/dlc_processor.py

+    """
+    Validate pose output shape and dtype.
+
+    Accepted runner output shapes:
+    - (K, 3): single-animal
+    - (N, K, 3): multi-animal
+    """


PR description mentions enforcing "value constraints" on pose outputs, but validate_pose_array currently only enforces shape and numeric dtype (it doesn’t check for finite values like NaN/Inf). Either add a finiteness check here (to match the stated contract) or update the stated contract/tests to reflect that NaN/Inf are allowed.

deruyter92 changed the title ~~Jaap/add compatibility tests~~ Add compatibility tests for DeepLabCut-live Feb 27, 2026

deruyter92 changed the title ~~Add compatibility tests for DeepLabCut-live~~ Add validation + compatibility tests for DeepLabCut-live Feb 27, 2026

C-Achard approved these changes Feb 27, 2026

View reviewed changes

C-Achard reviewed Feb 27, 2026

View reviewed changes

C-Achard mentioned this pull request Feb 27, 2026

Upgrade PySide6 multi-camera GUI #38

Merged

20 tasks

deruyter92 and others added 6 commits February 27, 2026 16:09

dlc_processor: add lightweight validation of shape and dtype (+tests)

41db059

Add basic compatibility tests for deeplabcut-live API

4c97d58

Add optional smoke test for exported dlclive model via env vars

1ad9a61

Update dlclivegui/services/dlc_processor.py

7e4b753

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

Update dlclivegui/services/dlc_processor.py

d9c0fdf

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

Update dlclivegui/services/dlc_processor.py

693c931

Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>

C-Achard force-pushed the jaap/add_compatibility_tests branch from 54ee55e to 693c931 Compare February 27, 2026 15:09

C-Achard reviewed Feb 27, 2026

View reviewed changes

.github/workflows/testing-ci.yml Outdated Show resolved Hide resolved

C-Achard added 2 commits February 27, 2026 16:14

Use Python 3.12 in CI testing matrix

60a0e8a

Update the GitHub Actions testing matrix to run on Python 3.12 instead of 3.11. This moves CI to test against the newer Python runtime while keeping existing matrix include entries unchanged.

Run pre-commit

990654b

C-Achard added 5 commits March 2, 2026 09:25

Fix hard-coded str to use enum

34b8727

C-Achard added DLClive Related to DLCLive inference, versioning, model export... CI Related to CI, tests, workflows... labels Mar 2, 2026

C-Achard assigned deruyter92 Mar 2, 2026

C-Achard added this to the PySide6 & multi-camera setup (2.0rc0 ?) milestone Mar 2, 2026

C-Achard marked this pull request as ready for review March 2, 2026 09:31

C-Achard self-requested a review March 2, 2026 09:31

C-Achard approved these changes Mar 2, 2026

View reviewed changes

C-Achard mentioned this pull request Mar 2, 2026

Several fixes to worker logic #57

Open

1 task

deruyter92 requested a review from Copilot March 11, 2026 11:09

Copilot started reviewing on behalf of deruyter92 March 11, 2026 11:10 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Conversation

deruyter92 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

C-Achard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deruyter92 commented Feb 27, 2026

Uh oh!

C-Achard commented Feb 27, 2026

Uh oh!

Uh oh!

deruyter92 commented Feb 28, 2026

Uh oh!

C-Achard commented Mar 2, 2026

Uh oh!

C-Achard commented Mar 2, 2026

Uh oh!

C-Achard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deruyter92 commented Feb 27, 2026 •

edited

Loading

C-Achard left a comment •

edited

Loading

C-Achard left a comment •

edited

Loading