0.65.0 (Draft)#

Draft — GitHub Release

Changes#

⚠️ Breaks#

Update DeltaArray to always pad values to 1024 elements and use bit transpose functions (#7007) @robert3005

✨ Features#

feat[array]: change VTable execution to pass owned arrays (#6867) @joseph-isaacs
feat(cuda): hybrid GPU dispatch - fuse dyn + standalone kernels (#7127) @0ax1
default mimalloc for vortex-python wheels (#7096) @onursatici
feat(cuda): add SEQUENCE source op to dyn dispatch (#7078) @0ax1
Vector Extension Type (#6964) @connortsui20
Canonical variant array (#7026) @AdamGS
IsSorted AggregateFn (#7029) @gatesn
IsConstant Aggregate Function (#7028) @gatesn
MinMax AggregateFn (#7006) @gatesn
Variant DType (#6912) @AdamGS
feat[back-compat]: Add a script to upload fixtures to a repo (#6993) @joseph-isaacs
NanCount AggregateFn (#7003) @gatesn
Cut over sum aggregate function (#6910) @gatesn
feat[compat-gen]: add expected_encodings to validate fixture construction (#6978) @joseph-isaacs

🚀 Performance#

do not validate already valid arrow varbin arrays (#7089) @onursatici
perf[fsst]: like pushdown using a dfa (#6935) @joseph-isaacs
Decimal sum uses smallest physical type that supports output dtype (#7008) @robert3005
perf: skip allocation for prim cast if possible (#6997) @0ax1
Faster true count using AVX2 and AVX512 instructions (#6931) @robert3005

🐛 Bug Fixes#

16 changes

fix[delta]: compress with nulls undef values use last valid value (or 0) (#7124) @joseph-isaacs
fix fuzzer issue #7074 (#7077) @joseph-isaacs
fix[back-compat]: use superset not subset in the check mod (#7075) @joseph-isaacs
fix[scalar_fns]: add correct pre-condition checks for all custom pushdown (#6977) @joseph-isaacs
Sum aggregate doesn’t include nan values (#7009) @robert3005
Fix scalar partial ordering comparison (#6999) @connortsui20
fix: compat script (#7024) @joseph-isaacs
fix[compat]: Update the action to use the correct check command (#7016) @joseph-isaacs
Add backwards compatibly handling of sum stats of extension types (#7014) @AdamGS
fix[vortex-array]: handle lazily sliced REE to_arrow (#7011) @asubiotto
Check for overflow in decimal sum (#7005) @gatesn
fix[vortex-array]: fix rebuild_trim_elements overflow (#6995) @asubiotto
fix: Zstd canonicalize with >2GB decompressed buffers (#6989) @SumedhArani
Sparse followup (#6980) @connortsui20
fix[sparse]: correctly handle cast for all-patched (#6979) @joseph-isaacs
fix[duckdb]: build cpp.rs only if changed (#6951) @joseph-isaacs

📖 Documentation#

More detailed docs on ScalarFnVTable (#7027) @connortsui20
Add high-level docs to vortex-btrblocks (#6984) @connortsui20

🧰 Maintenance#

52 changes

Testing FFI from C side (#7126) @myrrc
chore[release]: add compat-gen-upload to release (#7067) @joseph-isaacs
chore(deps): lock file maintenance (#7121) @renovate[bot]
chore(deps): update datafusion monorepo to v52.4.0 (#7120) @renovate[bot]
chore(deps): update all patch updates (#7119) @renovate[bot]
chore(deps): update taiki-e/cache-cargo-install-action digest to 1ce845d (#7117) @renovate[bot]
chore(deps): update codspeedhq/action digest to 1c8ae48 (#7116) @renovate[bot]
Only pass vtable once via ExtDType (#7105) @gatesn
Function to return vtable from an array (#7085) @gatesn
chore(deps): lock file maintenance (#7088) @renovate[bot]
Remove vortex-test-e2e (#7102) @connortsui20
bump aws-lc-rs to clear advisory (#7100) @onursatici
chore(deps): update rust crate tar to v0.4.45 [security] (#7099) @renovate[bot]
chore[mask]: intersect_by_rank benchmarks (#7083) @joseph-isaacs
chore: cache cpu feat check for micro-benchmarks (#7079) @0ax1
chore: AggregateFnRef [de]serialize (#7070) @blaginin
chore[fuzz]: (temp) remove duplicate struct field names (#7073) @joseph-isaacs
Update CLAUDE.md (#7069) @gatesn
Print to stdout in compress-bench and random-access-bench (#7065) @connortsui20
chore(deps): update rust crate oneshot to 0.2.0 (#7042) @renovate[bot]
chore(deps): update rust crate termtree to v1 (#7058) @renovate[bot]
chore(deps): update rust crate rand_distr to 0.6 (#7046) @renovate[bot]
chore(deps): update rust crate pyo3 to 0.28.0 (#7043) @renovate[bot]
chore(deps): update rust crate zip to v8 (#7059) @renovate[bot]
chore(deps): update ssciwr/doxygen-install action to v2 (#7060) @renovate[bot]
chore(deps): update actions/download-artifact action to v8 (#7056) @renovate[bot]
chore(deps): update aws-actions/configure-aws-credentials action to v6 (#7057) @renovate[bot]
chore(deps): update actions/checkout action to v6 (#7055) @renovate[bot]
chore(deps): update rust crate test-with to 0.15 (#7050) @renovate[bot]
chore(deps): update rust crate strum to 0.28 (#7048) @renovate[bot]
chore(deps): update rust crate sysinfo to 0.38.0 (#7049) @renovate[bot]
chore(deps): update rust crate sketches-ddsketch to 0.4.0 (#7047) @renovate[bot]
chore(deps): update rust crate noodles-vcf to 0.86.0 (#7041) @renovate[bot]
chore(deps): update plugin com.palantir.consistent-versions to v3.15.0 (#7035) @renovate[bot]
chore(deps): update rust crate noodles-bgzf to 0.46.0 (#7040) @renovate[bot]
chore(deps): update rust crate criterion to 0.8 (#7037) @renovate[bot]
chore(deps): update plugin com.vanniktech.maven.publish to v0.36.0 (#7036) @renovate[bot]
chore(deps): update plugin com.gradleup.shadow to v9.4.0 (#7034) @renovate[bot]
chore(deps): update plugin com.diffplug.spotless to v8.4.0 (#7033) @renovate[bot]
chore(deps): update gradle to v9.4.0 (#7032) @renovate[bot]
chore(deps): update rust crate fsst-rs to v0.5.8 (#7031) @renovate[bot]
chore(deps): update rust crate fsst-rs to v0.5.7 (#7030) @renovate[bot]
chore[ci]: use pre-build image and strip workflow installs (#7013) @joseph-isaacs
chore: clean up #[inline] annotations (#7000) @0ax1
chore[compat]: more fixtures compressed (#6986) @joseph-isaacs
chore(deps): update plugin com.palantir.git-version to v5 (#6949) @renovate[bot]
add more encoding fixtures (#6976) @joseph-isaacs
runtime: Instrument with tracing (#6981) @brancz
Remove git hooks example scripts (#6982) @connortsui20
Ignore justfile (#6941) @connortsui20
chore: only ignore c++ diagnostics for ddb headers (#6970) @0ax1
Fix fuzzer issue template (#6963) @connortsui20