DuckDB

Vortex extension is available from DuckDB 1.4.2+ on Linux and macOS (amd64, arm64). Windows support is planned.

Setup

INSTALL vortex; LOAD vortex;

Reading files

SELECT * FROM 'data.vortex';
# this syntax supports arguments like hive_partitioning=true
SELECT * FROM read_vortex('data.vortex');

Writing files

COPY (SELECT * FROM my_table) TO 'output.vortex';

Make sure to call LOAD vortex before writing any files. Duckdb writes CSV files by default, so if Vortex extension wasn’t loaded prior to COPY, output.vortex will be a CSV file. If you want to prevent this from happening, you can use COPY ... TO 'output.vortex' (FORMAT vortex) syntax instead which will fail if Vortex is not loaded.

Python client

import duckdb

duckdb.sql("INSTALL vortex")
duckdb.sql("LOAD vortex")

result = duckdb.sql("SELECT * FROM 'data.vortex' WHERE age > 30")
result.show()

Object storage secrets

Vortex does not use DuckDB’s secret storage. It uses environment variables via object_store instead.

If you want to read or write from an S3 bucket, set the following variables:

export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export AWS_REGION="your_bucket_region"

For Google Cloud Storage, use a key file variable:

export GOOGLE_APPLICATION_CREDENTIALS="my service account JSON key file"

For Azure Blob Storage, use

export AZURE_STORAGE_ACCOUNT="my account"
export AZURE_STORAGE_KEY="my key"