Vortex FFI API

Warning

This API should be considered entirely unstable. It _will_ change. Please reach out if a stable FFI API is important for your use-case and we can accelerate the process of stabilizing it.

enum vx_log_level

Log levels for the Vortex library.

type vx_dtype

The logical types of elements in Vortex arrays.

Vortex arrays preserve a single logical type, while the encodings allow for multiple physical ways to encode that type.

type vx_array

The FFI interface for an [Array].

Because dyn Trait pointers cannot be shared across FFI, we create a new struct to hold the wide pointer. The C FFI only seems a pointer to this structure, and can pass it into one of the various vx_array_* functions.

type vx_array_iter

The FFI interface for an [ArrayIterator].

type vx_array_sink

The sink interface is used to collect array chunks and place them into a resource (e.g. an array stream or file (vx_array_sink_open_file)).

type vx_error

The error structure populated by fallible Vortex C functions.

type vx_file_reader

A file reader that can be used to read from a file.

type vx_layout_reader

A Vortex layout reader.

struct vx_file_open_options

Options supplied for opening a file.

const char *uri

URI for opening the file. This must be a valid URI, even for files (file:///path/to/file)

const char *const *property_keys

Additional configuration for the file source (e.g. “s3.accessKey”). This may be null, in which case it is treated as empty.

const char *const *property_vals

Additional configuration values for the file source (e.g. S3 credentials).

int property_len

Number of properties in property_keys and property_vals.

struct vx_file_statistics

Whole file statistics.

uint64_t num_rows

The exact number of rows in the file.

struct vx_file_scan_options

Scan options provided by an FFI client calling the vx_file_scan function.

const char *const *projection

Column names to project out in the scan. These must be null-terminated C strings.

unsigned int projection_len

Number of columns in projection.

const char *filter_expression

Serialized expressions for pushdown

unsigned int filter_expression_len

The len in bytes of the filter expression

int split_by_row_count

Splits the file into chunks of this size, if zero then we use the write layout.

unsigned long row_range_start

First row of a range to scan.

unsigned long row_range_end

Last row of a range to scan.

struct vx_array *vx_array_iter_next(struct vx_array_iter *iter, struct vx_error **error)

Attempt to advance the current pointer of the iterator.

A return value of true indicates that another element was pulled from the iterator, and a return of false indicates that the iterator is finished.

It is an error to call this function again after the iterator is finished.

uint64_t vx_array_len(const struct vx_array *array)

Get the length of the array.

const struct vx_dtype *vx_array_dtype(const struct vx_array *array)

Get a pointer to the data type for an array.

Note that this pointer is tied to the lifetime of the array, and the caller is responsible for ensuring that it is never dereferenced after the array has been freed.

void vx_array_free(struct vx_array *array)

Free the array and all associated resources.

void vx_array_get_utf8(const struct vx_array *array, uint32_t index, void *dst, int *len)

Write the UTF-8 string at index in the array into the provided destination buffer, recording the length in len.

void vx_array_get_binary(const struct vx_array *array, uint32_t index, void *dst, int *len)

Write the UTF-8 string at index in the array into the provided destination buffer, recording the length in len.

struct vx_dtype *vx_dtype_new(uint8_t variant, bool nullable)

Pointer to a DType value that has been heap-allocated. Create a new simple dtype.

struct vx_dtype *vx_dtype_new_list(struct vx_dtype *element, bool nullable)

Create a new List type with the provided element type.

Upon successful return, this function moves the value out of the provided element pointer, so it is not safe to reference afterward.

void vx_dtype_free(struct vx_dtype *dtype)

Free an [DType] and all associated resources.

uint8_t vx_dtype_get(const struct vx_dtype *dtype)

Get the dtype variant tag for an [DType].

uint32_t vx_dtype_field_count(const struct vx_dtype *dtype)

For DTYPE_STRUCT variant DTypes, get the number of fields.

struct vx_dtype *vx_dtype_field_dtype(const struct vx_dtype *dtype, uint32_t index)

Get the dtype of a field in a DTYPE_STRUCT variant DType.

This returns a new owned, allocated copy of the DType that must be freed subsequently by the caller.

const struct vx_dtype *vx_dtype_element_type(const struct vx_dtype *dtype, struct vx_error **error)

For a list DType, get the inner element type.

The pointee’s lifetime is tied to the lifetime of the list DType. It should not be accessed after the list DType has been freed.

Converts a DType into a duckdb

Converts a DuckDB type into a vortex type

Back a single chunk of the array as a duckdb data chunk. The initial call should pass offset = 0. The offset is returned to the caller, which can be used to request the next chunk. 0 is returned when the stream is finished.

Pushed a single duckdb chunk into a file sink.

int vx_error_get_code(struct vx_error *error)

Return the integer error code from the given Vortex error.

const char *vx_error_get_message(struct vx_error *error)

Passes out an unowned reference to the error message from the given Vortex error. Return value is the length of the message string.

struct vx_file_reader *vx_file_open_reader(const struct vx_file_open_options *options, struct vx_error **error)

Open a file at the given path on the file system.

struct vx_dtype *vx_file_dtype(const struct vx_file_reader *file)

Get the DType of the data inside of the file.

struct vx_array_iter *vx_layout_reader_scan(const struct vx_layout_reader *layout_reader, const struct vx_file_scan_options *opts, struct vx_error **error)

Build a new vx_array_iter that returns a series of vx_array`s from a scan over a `vx_layout_reader.

uint64_t vx_file_row_count(struct vx_file_reader *file_reader, struct vx_error **error)

Returns the row count for a given file reader.

struct vx_layout_reader *vx_layout_reader_create(struct vx_file_reader *file_reader, struct vx_error **error)

Creates a layout reader for a given file.

void vx_file_reader_free(struct vx_file_reader *file)

Free the file and all associated resources.

This function will not automatically free any vx_array_iter() that were built from this file.

void vx_init_logging(enum vx_log_level level)

Initialize native logging with the specified level.

This function is optional, if it is not called then no runtime logger will be installed.

struct vx_array_sink *vx_array_sink_open_file(const char *path, const struct vx_dtype *dtype, struct vx_error **error)

Opens a writable array stream, where sink is used to push values into the stream. To close the stream close the sink with vx_array_sink_close.

void vx_array_sink_push(struct vx_array_sink *sink, const struct vx_array *array, struct vx_error **error)

Pushed a single array chunk into a file sink.

void vx_array_sink_close(struct vx_array_sink *sink, struct vx_error **error)

Closes an array sink, must be called to ensure all the values pushed to the sink are written to the external resource.