DuckDB

Vortex extension is available from DuckDB 1.4.2+ on Linux and macOS (amd64, arm64).

Setup

INSTALL vortex;
LOAD vortex;

Reading files

SELECT * FROM 'data.vortex';

Filters and projections are pushed down into Vortex, so only the columns and rows needed by the query are read and decompressed.

SELECT name, age FROM 'data.vortex' WHERE age > 30;

Writing files

Export data to Vortex using the COPY statement.

COPY (SELECT * FROM my_table) TO 'output.vortex';

Python

The DuckDB Python client works the same way:

import duckdb

duckdb.sql("INSTALL vortex")
duckdb.sql("LOAD vortex")

result = duckdb.sql("SELECT * FROM 'data.vortex' WHERE age > 30")
result.show()