Zarrquet implementation: Adding Zarrs-based TableProvider with bindings in Python. - #69

Closed

alxmrs wants to merge 65 commits into

mainfrom

Commits on Jul 15, 2025

Integrated rust into the project via maturin
alxmrs
committed
6f9fd56

Commits on Jul 16, 2025

Properly set up the rust project (with dummy table provide, can compile with `uv build`.
alxmrs
committed
fd6fe0e
Phase 1 and 2 implementation of zarr store plan!
alxmrs
committed
4dcaa52
Next step: only processes a single array in the zarr group as record batches
alxmrs
committed
3aef11a
Attempting to minimize copying.
alxmrs
committed
ff75797
Had Claude refactor the code to introduce generics, in order to reduce duplicate logic.
alxmrs
committed
3881787
Better explained requirements for tables, claude wrote tests.
alxmrs
committed
cbfc0e6
Applied automatic Zarr fixes from the compiler.
alxmrs
committed
97e7a3c
Core data var to table logic implemented, according to Claude. Needs some refactoring.
alxmrs
committed
baf4f3b
Adding test_data to gitignore.
alxmrs
committed
09251fb
Added example and test scripts as sanity checks. might delete later.
alxmrs
committed
df8debd
Adding predicate pushdown logic, but I think it may need revision, will investigate later.
alxmrs
committed
b55db35

Commits on Jul 17, 2025

Generated integration tests to test sql functionality.
alxmrs
committed
b6efef2
Initial `from_zarr` implementation, which calls the rust TableProvider, with unit tests.
alxmrs
committed
9e6a0b0
First stage of multi-stage test refactoring from Claude: creating a test base.
alxmrs
committed
1937393
Made some minor refactors.
alxmrs
committed
6457dd8
Implemented strategy 2 and 3.
alxmrs
committed
0777064
Import opt.
alxmrs
committed
7a5700a
Tests for semantic equivalence to xarray operations.
alxmrs
committed
624c3af
Moved test data creation scripts into examples.
alxmrs
committed
7c328e0
Removing examples for now, because they add too much noise to the review.
alxmrs
committed
028e69b
Reformatted the python code with pyink.
alxmrs
committed
b30bde6
Import optimization
alxmrs
committed
fff4203
Revert to simpler tests! These tests have gotten too complicated and illegible.
alxmrs
committed
0dd67fd
WIP, manually refactoring tests to test both from_zarr and from_dataset
alxmrs
committed
cd59d51
I think I found an actual error with the Zarr logic.
alxmrs
committed
27baf67
Fixed signficant correctness bugs in the tabel provider impl related to corrent column names as well as correct scanning (type handing).
alxmrs
committed
38f2796
Test is incorrect; updating.
alxmrs
committed
fdf19bf
Table provider passes tests, but it looks like it needs to be simplified!
alxmrs
committed
8476f5f
Continuing my zarr vs dataset refactor.
alxmrs
committed
76196d6
Pinning down the first of a two part problem.
alxmrs
committed
c2d95c2
Added skips to tests to look into later.
alxmrs
committed
acbb2bb
Reformatted tests.
alxmrs
committed
6e02906

Commits on Jul 18, 2025

Attempt to fix minor warnings.
alxmrs
committed
c85d382

Commits on Jul 22, 2025

Commits on Jul 24, 2025

Apply clippy suggestions and rustfmt to #69 (#70)
kylebarron
authored
8b6beb5

Commits on Jul 25, 2025

Commits on Jul 27, 2025

`from_zarr` is now only supported for Python 3.11+
alxmrs
committed
7a3ca51
pyink fmt
alxmrs
committed
43e5046
Try to fix linker problem for the maturin arm build.
alxmrs
committed
92203fe
Put in the wrong file
alxmrs
committed
7e20540
Upgrade ubuntu version to get gcc 14
alxmrs
committed
29d132e
Attempting to fix IBM target via: https://github.com/rust-lang/stacker/issues/79#issuecomment-1497479267
alxmrs
committed
986b498

Commits on Jul 29, 2025

Add test to create test datasets for rust tests.
alxmrs
committed
ea6870a
Adding integration tests from Claude.
alxmrs
committed
b7a8494
Adding build step to run integration tests
alxmrs
committed
69f065a
fmt int tests
alxmrs
committed
243015b

Commits on Sep 11, 2025

Attempting to take a much simpler approach via an adapter.
alxmrs
committed
14b4ea2

Commits on Sep 12, 2025

New, simpler approach that makes use of `arrow-zarr`.
alxmrs
committed
5ac86e4

Commits on Sep 26, 2025

Current WIP, got back to where I started :/
alxmrs
committed
82e54cc
Fix: don't pass chunks.
alxmrs
committed
7f24256

Commits on Sep 27, 2025