0.76.0 (Draft)#

Draft — GitHub Release

Changes#

⚠️ Breaks#

Fix bounded max partial state (#8515) @gatesn
Use aggregate descriptors for zoned stats (#7938) @gatesn
Remove duckdbfs (#8468) @myrrc
Move execute-parent kernels into session registry (#8482) @gatesn
break: add Validity::definitely_all_null() method (#8447) @joseph-isaacs
Make Mask execution strict about nullable booleans (#8121) @joseph-isaacs
Support reporting statistics in spark datasource (#8057) @robert3005

✨ Features#

Decimal casting (#8562) @gatesn
Add JSON to Parquet Variant conversion with shredding (#8391) @AdamGS
Add extension storage scalar function (#8540) @gatesn
Numerical aggregate functions have an option to skip or include nans in calculation, skip by default (#8457) @robert3005
feat(vortex-geo): ST_Distance via the geo crate + Polygon type (#8497) @HarukiMoriarty
feat[gpu]: arrow device array stream support (#8483) @0ax1
feat(vortex-datafusion): struct scalar conversion + extension-over-struct scan (#8453) @HarukiMoriarty
feat: DeltaScheme adjustable compression ratio threshold (#8461) @joseph-isaacs
Add Delta scheme to the integer compressor (unstable_encodings) (#8212) @joseph-isaacs
Push down some expressions to Dict layout reader’s cached values (#8341) @myrrc
–patterns flag for random-access-bench (#8446) @myrrc
ffi: create data source from memory buffer (#8426) @myrrc
Constant comparison and byte_length OnPair kernels (#8371) @myrrc
REGEX support for sqllogictests (#8392) @myrrc
feat(vortex-geo): Arrow import/export for the native Point type (#8374) @HarukiMoriarty

🚀 Performance#

Optimize interleave boolean gather (#8350) @joseph-isaacs
RLEExporter for duckdb (#8539) @myrrc
ReusableDict for RunEnd duckdb exporter (#8538) @myrrc
perf: align buffer to Alignment::DEFAULT_ALIGNMENT = Alignment::new(256) (#8490) @joseph-isaacs
DuckDB chunk exporter (#8520) @gatesn
Faster take for runend array (#8228) @robert3005
Inline Kleene boolean kernels (#8478) @gatesn
Scalar functions pushdown to Duckdb (#8303) @myrrc
perf: don’t use fill_null for executing an array into a mask (#8466) @joseph-isaacs
Optimize buffer ops (#8322) @AdamGS
perf[gpu]: export arrow device validity on the gpu (#8440) @0ax1
perf[buffer]: iteration for fallible operations with validity (#8120) @joseph-isaacs
Faster str interning (#8459) @AdamGS
perf[gpu]: generate FixedSizeList offsets on device (#8458) @0ax1
Arithmetic Operations (#8398) @gatesn
perf(buffer): read byte-aligned bit operands directly as words (#8436) @miniex
perf: non-blocking Arrow Device array release (#8390) @0ax1

🐛 Bug Fixes#

12 changes

fix(layout): don’t panic collecting an empty stream (#8472) @miniex
Correctly calculate FSST compressed output size (#8551) @robert3005
refactor[fsst]: take ArrayRef in fsst_compress, decide offset width upfront (#7900) @mprammer
Move optimizer kernels into a dedicated KernelSession (#8511) @gatesn
Fix is_zero() for FixedSizeList and Struct scalar types (#8500) @connortsui20
Handle null lanes in primitive take (#8505) @gatesn
Preserve dict referenced values metadata (#8499) @gatesn
fix: preserve operand width in DecimalValue checked arithmetic (#7380) @abnobdoss
fix Fuzz constant OOM for CUDA (#8481) @a10y
vortex-tui: query tab should work for non-*.vortex files (#8473) @a10y
Ignore PyO3 rustsec (#8381) @connortsui20
fix: vx_data_source_new_buffer sem-merge conflict (#8437) @joseph-isaacs

📖 Documentation#

Expanding conceptual docs (and some other minor docs) (#8552) @AdamGS
docs: update doc links (#8375) @joseph-isaacs

🧰 Maintenance#

38 changes

Refactor DictEncoder to not use ArrayAccessors (#7759) @robert3005
Clear 3 orphan json files from root of repo (#8558) @AdamGS
Remove LEGACY_SESSION from benchmarks and back compat tests (#8554) @robert3005
Bump codspeed to 5.0 (#8553) @robert3005
Remove usages of LEGACY_SESSION from tests (#8547) @robert3005
assert_arrays_eq and assert_nth_scalar require ExecutionCtx (#8509) @robert3005
Update Rust crate memmap2 to v0.9.11 [SECURITY] (#8545) @renovate[bot]
Update lance benchmark dependencies (major) (#8525) @renovate[bot]
Update actions/setup-java digest to ad2b381 (#8521) @renovate[bot]
Update anthropics/claude-code-action digest to 2fee155 (#8522) @renovate[bot]
Update taiki-e/install-action digest to 9e1e580 (#8523) @renovate[bot]
Update actions/checkout action to v7 (#8524) @renovate[bot]
Lock file maintenance JS lock file maintenance (#8536) @renovate[bot]
Lock file maintenance Rust lock file maintenance (#8537) @renovate[bot]
Make sure both pushdown features are always enabled for DF benchmarks (#8507) @AdamGS
Misc upgrades to VortexSource (#8501) @AdamGS
Generalize SIMD take operations to Copy (#8496) @connortsui20
Migrate to new stats falsification rules (#8345) @gatesn
codegen only used functions in duckdb C API (#8491) @myrrc
Change ALIGNMENT_TO_HOST_COPY to Alignment::HOST_COPY (#8488) @AdamGS
Touch up some of the interfaces and docs in vortex-datafusion (#8485) @AdamGS
Add benchmarks for take on runend array (#8469) @robert3005
Include vortex-compute in codspeed benchmarks (#8477) @robert3005
Add Kleene boolean benchmarks (#8476) @gatesn
Immutable session backed by HashMap (#8454) @AdamGS
test[gpu]: expand cuDF CUDA e2e coverage (#8463) @0ax1
Upgrade pyO3 to 0.29.0 (#8462) @robert3005
Update dependency starlette to v1.3.1 [SECURITY] (#8435) @renovate[bot]
fmt: normalize_comments = true (#8428) @joseph-isaacs
Update taiki-e/install-action digest to 7a79fe8 (#8409) @renovate[bot]
Update gradle/actions digest to 3f131e8 (#8408) @renovate[bot]
Update anthropics/claude-code-action digest to d5726de (#8407) @renovate[bot]
Update dependency duckdb to v1.4.2 [SECURITY] (#8431) @renovate[bot]
Update dependency pip to v26.1.2 [SECURITY] (#8432) @renovate[bot]
Align data buffers in fastlanes benchmarks (#8393) @AdamGS
Update plugin com.palantir.java-format to v2.93.0 (#8416) @renovate[bot]
More arithmetic benchmarks (#8402) @gatesn
Add binary operator benchmarks (#8396) @gatesn