Vortex FFI API

Warning

This API should be considered entirely unstable. It _will_ change. Please reach out if a stable FFI API is important for your use-case and we can accelerate the process of stabilizing it.

enum vx_log_level

Log levels for the Vortex library.

type vx_dtype

The logical types of elements in Vortex arrays.

Vortex arrays preserve a single logical type, while the encodings allow for multiple physical ways to encode that type.

type vx_array

The FFI interface for an [Array].

Because dyn Trait pointers cannot be shared across FFI, we create a new struct to hold the wide pointer. The C FFI only seems a pointer to this structure, and can pass it into one of the various vx_array_* functions.

type vx_array_stream

FFI-exposed stream interface.

type vx_error

The error structure populated by fallible Vortex C functions.

type vx_file_reader

A file reader that can be used to read from a file.

struct vx_file_open_options

Options supplied for opening a file.

const char *uri

URI for opening the file. This must be a valid URI, even for files (file:///path/to/file)

const char *const *property_keys

Additional configuration for the file source (e.g. “s3.accessKey”). This may be null, in which case it is treated as empty.

const char *const *property_vals

Additional configuration values for the file source (e.g. S3 credentials).

int property_len

Number of properties in property_keys and property_vals.

struct vx_file_statistics

Whole file statistics.

uint64_t num_rows

The exact number of rows in the file.

struct vx_file_scan_options

Scan options provided by an FFI client calling the vx_file_scan function.

const char *const *projection

Column names to project out in the scan. These must be null-terminated C strings.

int projection_len

Number of columns in projection.

int split_by_row_count

Splits the file into chunks of this size, if zero then we use the write layout.

uint64_t vx_array_len(const struct vx_array *array)

Get the length of the array.

const struct vx_dtype *vx_array_dtype(const struct vx_array *array)

Get a pointer to the data type for an array.

Note that this pointer is tied to the lifetime of the array, and the caller is responsible for ensuring that it is never dereferenced after the array has been freed.

void vx_array_free(struct vx_array *array)

Free the array and all associated resources.

void vx_array_get_utf8(const struct vx_array *array, uint32_t index, void *dst, int *len)

Write the UTF-8 string at index in the array into the provided destination buffer, recording the length in len.

void vx_array_get_binary(const struct vx_array *array, uint32_t index, void *dst, int *len)

Write the UTF-8 string at index in the array into the provided destination buffer, recording the length in len.

struct vx_dtype *vx_dtype_new(uint8_t variant, bool nullable)

Pointer to a DType value that has been heap-allocated. Create a new simple dtype.

struct vx_dtype *vx_dtype_new_list(struct vx_dtype *element, bool nullable)

Create a new List type with the provided element type.

Upon successful return, this function moves the value out of the provided element pointer, so it is not safe to reference afterward.

void vx_dtype_free(struct vx_dtype *dtype)

Free an [DType] and all associated resources.

uint8_t vx_dtype_get(const struct vx_dtype *dtype)

Get the dtype variant tag for an [DType].

uint32_t vx_dtype_field_count(const struct vx_dtype *dtype)

For DTYPE_STRUCT variant DTypes, get the number of fields.

struct vx_dtype *vx_dtype_field_dtype(const struct vx_dtype *dtype, uint32_t index)

Get the dtype of a field in a DTYPE_STRUCT variant DType.

This returns a new owned, allocated copy of the DType that must be freed subsequently by the caller.

const struct vx_dtype *vx_dtype_element_type(const struct vx_dtype *dtype, struct vx_error **error)

For a list DType, get the inner element type.

The pointee’s lifetime is tied to the lifetime of the list DType. It should not be accessed after the list DType has been freed.

Converts a DType into a duckdb

Back a single chunk of the array as a duckdb data chunk. The initial call should pass offset = 0. The offset is returned to the caller, which can be used to request the next chunk. 0 is returned when the stream is finished.

int vx_error_get_code(struct vx_error *error)

Return the integer error code from the given Vortex error.

const char *vx_error_get_message(struct vx_error *error)

Passes out an unowned reference to the error message from the given Vortex error. Return value is the length of the message string.

struct vx_file_reader *vx_file_open_reader(const struct vx_file_open_options *options, struct vx_error **error)

Open a file at the given path on the file system.

const struct vx_dtype *vx_file_dtype(const struct vx_file_reader *file)

Get a readonly pointer to the DType of the data inside of the file.

The pointer’s lifetime is tied to the lifetime of the underlying file, so it should not be dereferenced after the file has been freed.

struct vx_array_stream *vx_file_scan(const struct vx_file_reader *file, const struct vx_file_scan_options *opts, struct vx_error **error)

Build a new vx_array_stream that return a series of vx_array`s scan over a `vx_file.

void vx_file_reader_free(struct vx_file_reader *file)

Free the file and all associated resources.

This function will not automatically free any vx_array_stream() that were built from this file.

void vx_init_logging(enum vx_log_level level)

Initialize native logging with the specified level.

This function is optional, if it is not called then no runtime logger will be installed.

const struct vx_dtype *vx_array_stream_dtype(const struct vx_array_stream *stream)

Gets the dtype from an array stream, if the stream is finished the DType is null

struct vx_array *vx_array_stream_next(struct vx_array_stream *stream, struct vx_error **error)

Attempt to advance the current pointer of the stream.

A return value of true indicates that another element was pulled from the stream, and a return of false indicates that the stream is finished.

It is an error to call this function again after the stream is finished.

bool vx_array_stream_finished(const struct vx_array_stream *stream)

Predicate function to check if the array stream is finished.

void vx_array_stream_free(struct vx_array_stream *stream)

Free the array stream and all associated resources.