Index (vortex-spark_2.13 e3c7401.dirty API)

A B C D E F G H I M N O P R S T V W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form

A

abort() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
abort() - Method in class dev.vortex.spark.write.VortexDataWriter: Aborts the write operation and cleans up resources.
abort(WriterCommitMessage[]) - Method in class dev.vortex.spark.write.VortexBatchWrite: Aborts the write job due to failures.
accessKey() - Method in class dev.vortex.spark.config.VortexAzureProperties
accessKeyId() - Method in class dev.vortex.spark.config.VortexS3Properties
addAllColumns(Iterable<Column>) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds multiple columns to read.
addAllPaths(Iterable<String>) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds multiple file paths to scan.
addColumn(Column) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds a column to read.
addPath(String) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds a file path to scan.
ArrowUtils - Class in dev.vortex.spark: Utility class for converting Arrow types to Spark SQL data types.
asProperties() - Method in class dev.vortex.spark.config.VortexAzureProperties
asProperties() - Method in class dev.vortex.spark.config.VortexS3Properties
azurePropertiesFromHadoopConf(Configuration) - Static method in class dev.vortex.spark.config.HadoopUtils

B

build() - Method in class dev.vortex.spark.read.VortexScanBuilder: Builds a VortexScan with the configured paths and columns.
build() - Method in class dev.vortex.spark.write.VortexWriteBuilder: Builds a Write for executing the write operation.
bytesWritten() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the number of bytes written to the file.

C

capabilities() - Method in class dev.vortex.spark.VortexTable: Returns the capabilities supported by this table.
close() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Closes this column vector and releases any associated resources.
close() - Method in class dev.vortex.spark.read.VortexColumnarBatch: Closes this columnar batch and releases all associated resources.
close() - Method in class dev.vortex.spark.read.VortexColumnarBatchIterator: Closes this iterator and releases all associated resources.
close() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
close() - Method in class dev.vortex.spark.write.VortexDataWriter: Closes the writer and releases resources.
closeIfFreeable() - Method in class dev.vortex.spark.read.VortexColumnarBatch: Closes this columnar batch if it is freeable and releases all associated resources.
columnarSupportMode() - Method in class dev.vortex.spark.read.VortexScan: Returns the columnar support mode for this scan.
commit() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
commit() - Method in class dev.vortex.spark.write.VortexDataWriter: Commits the write operation and returns a commit message.
commit(WriterCommitMessage[]) - Method in class dev.vortex.spark.write.VortexBatchWrite: Commits the entire write job after all tasks complete successfully.
convert(StructType) - Static method in class dev.vortex.spark.write.SparkToArrowSchema: Converts a Spark StructType schema to an Arrow Schema.
createBatchWriterFactory(PhysicalWriteInfo) - Method in class dev.vortex.spark.write.VortexBatchWrite: Creates a DataWriterFactory for producing data writers on executors.
createConstantVector(int, DataType, String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Creates a Spark ConstantColumnVector populated with the given partition value, parsed according to the target DataType.
createReaderFactory() - Method in class dev.vortex.spark.read.VortexBatchExec: Creates a factory for creating partition readers.
createWriter(int, long) - Method in class dev.vortex.spark.write.VortexDataWriterFactory: Creates a new data writer for a specific partition and task.

D

description() - Method in class dev.vortex.spark.read.VortexScan: Logging-friendly readable description of the scan source.
dev.vortex.spark - package dev.vortex.spark
dev.vortex.spark.config - package dev.vortex.spark.config
dev.vortex.spark.read - package dev.vortex.spark.read
dev.vortex.spark.write - package dev.vortex.spark.write

E

endpoint() - Method in class dev.vortex.spark.config.VortexS3Properties
equals(Object) - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Indicates whether some other object is "equal to" this one.

F

filePath() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the path to the written Vortex file.
fromArrowField(Field) - Static method in class dev.vortex.spark.ArrowUtils: Converts an Arrow Field to a Spark SQL DataType.
fromArrowType(ArrowType) - Static method in class dev.vortex.spark.ArrowUtils: Converts an Arrow type to a Spark SQL DataType.

G

getArray(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the array value at the specified row.
getBinary(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the binary data (byte array) at the specified row.
getBoolean(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the boolean value at the specified row.
getByte(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the byte value at the specified row.
getChild(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the child column at the specified ordinal.
getColumns() - Method in class dev.vortex.spark.VortexFilePartition: Returns the list of columns to be read from this partition.
getDecimal(int, int, int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the decimal value at the specified row with the given precision and scale.
getDouble(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the double value at the specified row.
getFloat(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the float value at the specified row.
getFormatOptions() - Method in class dev.vortex.spark.VortexFilePartition
getInt(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the int value at the specified row.
getLong(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the long value at the specified row.
getMap(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the map value at the specified row.
getPartitionMessages() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter.PartitionedWriterCommitMessage: Returns the commit messages from each individual partition writer.
getPartitionValues() - Method in class dev.vortex.spark.VortexFilePartition: Returns the partition column values parsed from this file's Hive-style directory path.
getPath() - Method in class dev.vortex.spark.VortexFilePartition: Returns the file system path to the Vortex file for this partition.
getShort(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the short value at the specified row.
getTable(StructType, Transform[], Map<String, String>) - Method in class dev.vortex.spark.VortexDataSourceV2: Creates a Vortex table instance with the given schema and properties.
getUTF8String(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the UTF8String value at the specified row.
getValueVector() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the underlying Apache Arrow ValueVector wrapped by this column vector.

H

HadoopUtils - Class in dev.vortex.spark.config
hashCode() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Returns a hash code value for this object.
hasNext() - Method in class dev.vortex.spark.read.VortexColumnarBatchIterator: Returns whether there are more columnar batches available.
hasNull() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns whether this column contains any null values.

I

inferPartitionColumnType(String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Infers a Spark DataType from a partition value string.
inferSchema(CaseInsensitiveStringMap) - Method in class dev.vortex.spark.VortexDataSourceV2: Infers the schema of the Vortex files specified in the options.
isNullAt(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns whether the value at the specified row is null.

M

MAX_BUFFER_BYTES - Static variable in class dev.vortex.spark.read.VortexColumnarBatchIterator: Maximum buffer size in bytes for prefetching arrays.

N

name() - Method in class dev.vortex.spark.VortexTable: Returns the name of this table.
newScanBuilder(CaseInsensitiveStringMap) - Method in class dev.vortex.spark.VortexTable: Creates a new ScanBuilder for this table.
newWriteBuilder(LogicalWriteInfo) - Method in class dev.vortex.spark.VortexTable: Creates a new WriteBuilder for writing data to this table.
next() - Method in class dev.vortex.spark.read.VortexColumnarBatchIterator: Returns the next columnar batch from the iterator.
numNulls() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the total number of null values in this column.

O

onDataWriterCommit(WriterCommitMessage) - Method in class dev.vortex.spark.write.VortexBatchWrite: Called when a single data writer task completes successfully.

P

parsePartitionValues(String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Parses Hive-style key=value segments from a file path.
PartitionedVortexDataWriter - Class in dev.vortex.spark.write: Writes Spark InternalRow data to Vortex files organized in Hive-style partition directories.
PartitionedVortexDataWriter.PartitionedWriterCommitMessage - Class in dev.vortex.spark.write: Commit message that aggregates results from multiple partition writers.
partitioning() - Method in class dev.vortex.spark.VortexTable: Returns the partitioning transforms for this table.
PartitionPathUtils - Class in dev.vortex.spark.read: Utilities for discovering and materializing Hive-style partition columns from file paths.
planInputPartitions() - Method in class dev.vortex.spark.read.VortexBatchExec: Plans the input partitions for this batch scan.
pruneColumns(StructType) - Method in class dev.vortex.spark.read.VortexScanBuilder: Prunes the columns to only include those specified in the required schema.

R

readSchema() - Method in class dev.vortex.spark.read.VortexScan: Returns the schema for the data that will be read by this scan.
recordCount() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the number of records written to the file.
region() - Method in class dev.vortex.spark.config.VortexS3Properties

S

s3PropertiesFromHadoopConf(Configuration) - Static method in class dev.vortex.spark.config.HadoopUtils
sasKey() - Method in class dev.vortex.spark.config.VortexAzureProperties
schema() - Method in class dev.vortex.spark.VortexTable: Returns the schema of this table.
secretAccessKey() - Method in class dev.vortex.spark.config.VortexS3Properties
sessionToken() - Method in class dev.vortex.spark.config.VortexS3Properties
setAccessKey(String) - Method in class dev.vortex.spark.config.VortexAzureProperties
setAccessKeyId(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setEndpoint(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setRegion(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSasKey(String) - Method in class dev.vortex.spark.config.VortexAzureProperties
setSecretAccessKey(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSessionToken(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSkipSignature(boolean) - Method in class dev.vortex.spark.config.VortexAzureProperties
setSkipSignature(boolean) - Method in class dev.vortex.spark.config.VortexS3Properties
shortName() - Method in class dev.vortex.spark.VortexDataSourceV2: Returns the short name identifier for this data source.
skipSignature() - Method in class dev.vortex.spark.config.VortexAzureProperties
skipSignature() - Method in class dev.vortex.spark.config.VortexS3Properties
SparkToArrowSchema - Class in dev.vortex.spark.write: Utility class for converting Spark SQL schemas to Arrow schemas.
SparkTypes - Class in dev.vortex.spark: Helpers for converting between Spark and Vortex type systems.
supportsExternalMetadata() - Method in class dev.vortex.spark.VortexDataSourceV2: Indicates whether this data source supports external metadata (schemas).

T

toBatch() - Method in class dev.vortex.spark.read.VortexScan: Converts this scan to a Batch for execution.
toBatch() - Method in class dev.vortex.spark.write.VortexBatchWrite: Returns this object as a BatchWrite.
toColumns(DType) - Static method in class dev.vortex.spark.SparkTypes: Convert a STRUCT Vortex type to a Spark Column.
toDataType(DType) - Static method in class dev.vortex.spark.SparkTypes: Convert a STRUCT Vortex type to a Spark DataType.
toDType(StructType) - Static method in class dev.vortex.spark.SparkTypes
toString() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Returns a string representation of this record class.
truncate() - Method in class dev.vortex.spark.write.VortexWriteBuilder: Configures the write operation to truncate existing data.

V

VortexArrowColumnVector - Class in dev.vortex.spark.read: Spark ColumnVector implementation that wraps Apache Arrow vectors from Vortex data.
VortexArrowColumnVector(ValueVector) - Constructor for class dev.vortex.spark.read.VortexArrowColumnVector: Creates a new VortexArrowColumnVector wrapping the specified Arrow ValueVector.
VortexAzureProperties - Class in dev.vortex.spark.config
VortexAzureProperties() - Constructor for class dev.vortex.spark.config.VortexAzureProperties
VortexBatchExec - Class in dev.vortex.spark.read: Execution source for batch scans of Vortex file tables.
VortexBatchExec(ImmutableList<String>, ImmutableList<Column>, ImmutableMap<String, String>) - Constructor for class dev.vortex.spark.read.VortexBatchExec: Creates a new VortexBatchExec for scanning the specified Vortex files.
VortexBatchWrite - Class in dev.vortex.spark.write: Manages the batch write operation for creating Vortex files.
VortexColumnarBatch - Class in dev.vortex.spark.read: A ColumnarBatch that returns Vortex-managed memory with Arrow format, shared over the C Data Interface.
VortexColumnarBatch(Array, ColumnVector[], int) - Constructor for class dev.vortex.spark.read.VortexColumnarBatch: Creates a new VortexColumnarBatch with the specified backing array and column vectors.
VortexColumnarBatchIterator - Class in dev.vortex.spark.read: Iterator that converts Vortex Arrays into Spark ColumnarBatch objects.
VortexColumnarBatchIterator(ArrayIterator) - Constructor for class dev.vortex.spark.read.VortexColumnarBatchIterator: Creates a new VortexColumnarBatchIterator that wraps the given ArrayIterator.
VortexDataSourceV2 - Class in dev.vortex.spark: Spark V2 data source for reading and writing Vortex files.
VortexDataSourceV2() - Constructor for class dev.vortex.spark.VortexDataSourceV2: Creates a new instance of the Vortex data source.
VortexDataWriter - Class in dev.vortex.spark.write: Writes Spark InternalRow data to a Vortex file.
VortexDataWriterFactory - Class in dev.vortex.spark.write: Factory for creating VortexDataWriter instances on Spark executors.
VortexFilePartition - Class in dev.vortex.spark: An InputPartition for reading a whole Vortex file.
VortexFilePartition(String, ImmutableList<Column>, ImmutableMap<String, String>, ImmutableMap<String, String>) - Constructor for class dev.vortex.spark.VortexFilePartition: Creates a new Vortex file partition.
VortexS3Properties - Class in dev.vortex.spark.config
VortexS3Properties() - Constructor for class dev.vortex.spark.config.VortexS3Properties
VortexScan - Class in dev.vortex.spark.read: Spark V2 Scan over a table of Vortex files.
VortexScan(ImmutableList<String>, ImmutableList<Column>, ImmutableMap<String, String>) - Constructor for class dev.vortex.spark.read.VortexScan: Creates a new VortexScan for the specified file paths and columns.
VortexScanBuilder - Class in dev.vortex.spark.read: Spark V2 ScanBuilder for table scans over Vortex files.
VortexScanBuilder(Map<String, String>) - Constructor for class dev.vortex.spark.read.VortexScanBuilder: Creates a new VortexScanBuilder with empty paths and columns.
VortexTable - Class in dev.vortex.spark: Spark V2 Table of Vortex files that supports both reading and writing.
VortexTable(ImmutableList<String>, StructType, Map<String, String>, Transform[]) - Constructor for class dev.vortex.spark.VortexTable: Creates a new VortexTable with read/write support.
VortexWriteBuilder - Class in dev.vortex.spark.write: Builder for configuring Vortex write operations.
VortexWriteBuilder(String, LogicalWriteInfo, Map<String, String>, Transform[]) - Constructor for class dev.vortex.spark.write.VortexWriteBuilder: Creates a new VortexWriteBuilder.
VortexWriterCommitMessage - Record Class in dev.vortex.spark.write: Commit message containing information about a successfully written Vortex file.
VortexWriterCommitMessage(String, long, long) - Constructor for record class dev.vortex.spark.write.VortexWriterCommitMessage: Creates a new commit message for a written Vortex file.

W

write(InternalRow) - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
write(InternalRow) - Method in class dev.vortex.spark.write.VortexDataWriter: Writes a single row to the Vortex file.

A B C D E F G H I M N O P R S T V W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form