Skip to content

Conversation

@godnight10061
Copy link
Contributor

Summary

  • Add a component-shredded vortex.list layout for List/FixedSizeList columns
  • Enable nested projection/pushdown for list<struct> (e.g. projecting items.a)
  • Extend vortex.get_item and the list reader to support list<struct>

Why

  • Today list columns are written as opaque blobs, which prevents nested projections from being pushed down and breaks list-of-struct field selection.

Tests

  • cargo +nightly fmt --all --check
  • cargo clippy --locked --workspace --all-features --all-targets --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench -- -D warnings
  • cargo nextest run --locked --workspace --all-features --no-fail-fast --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench (fails locally: vortex-cuda for_::tests::test_cuda_for_decompression requires a CUDA driver)
  • cargo nextest run --locked --workspace --all-features --no-fail-fast --exclude vortex-cuda --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench

Compatibility

  • New files may be written with a new list layout encoding id (vortex.list). Older readers that don?t know this encoding won?t be able to read those files; new readers continue to read existing files.

References

@godnight10061
Copy link
Contributor Author

I wasn't able to apply labels from a fork (permission denied). Could a maintainer please add the ix label for the changelog/CI gating?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant