Index (vortex-spark_2.13 126c431.dirty API)

A B C D E F G H I N O P R S T V W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form

A

abort() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
abort() - Method in class dev.vortex.spark.write.VortexDataWriter: Aborts the write operation and cleans up resources.
abort(WriterCommitMessage[]) - Method in class dev.vortex.spark.write.VortexBatchWrite: Aborts the write job due to failures.
accessKey() - Method in class dev.vortex.spark.config.VortexAzureProperties
accessKeyId() - Method in class dev.vortex.spark.config.VortexS3Properties
addAllColumns(Iterable<Column>) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds multiple columns to read.
addAllPaths(Iterable<String>) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds multiple file paths to scan.
addColumn(Column) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds a column to read.
addPath(String) - Method in class dev.vortex.spark.read.VortexScanBuilder: Adds a file path to scan.
ArrowUtils - Class in dev.vortex.spark: Utility class for converting Arrow types to Spark SQL data types.
asProperties() - Method in class dev.vortex.spark.config.VortexAzureProperties
asProperties() - Method in class dev.vortex.spark.config.VortexS3Properties
azurePropertiesFromHadoopConf(Configuration) - Static method in class dev.vortex.spark.config.HadoopUtils

B

build() - Method in class dev.vortex.spark.read.VortexScanBuilder: Builds a VortexScan with the configured paths and columns.
build() - Method in class dev.vortex.spark.write.VortexWriteBuilder: Builds a Write for executing the write operation.
bytesWritten() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the number of bytes written to the file.

C

capabilities() - Method in class dev.vortex.spark.VortexTable: Returns the capabilities supported by this table.
close() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: No-op: the underlying Arrow ValueVectors are owned by the ArrowReader that produced this batch and are released when that reader is closed.
close() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
close() - Method in class dev.vortex.spark.write.VortexDataWriter: Closes the writer and releases resources.
columnarSupportMode() - Method in class dev.vortex.spark.read.VortexScan: Returns the columnar support mode for this scan.
commit() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
commit() - Method in class dev.vortex.spark.write.VortexDataWriter: Commits the write operation and returns a commit message.
commit(WriterCommitMessage[]) - Method in class dev.vortex.spark.write.VortexBatchWrite: Commits the entire write job after all tasks complete successfully.
convert(StructType) - Static method in class dev.vortex.spark.write.SparkToArrowSchema: Converts a Spark StructType schema to an Arrow Schema.
createBatchWriterFactory(PhysicalWriteInfo) - Method in class dev.vortex.spark.write.VortexBatchWrite: Creates a DataWriterFactory for producing data writers on executors.
createColumnarReader(InputPartition) - Method in class dev.vortex.spark.read.VortexPartitionReaderFactory
createConstantVector(int, DataType, String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Creates a Spark ConstantColumnVector populated with the given partition value, parsed according to the target DataType.
createReader(InputPartition) - Method in class dev.vortex.spark.read.VortexPartitionReaderFactory
createReaderFactory() - Method in class dev.vortex.spark.read.VortexBatchExec
createWriter(int, long) - Method in class dev.vortex.spark.write.VortexDataWriterFactory: Creates a new data writer for a specific partition and task.

D

description() - Method in class dev.vortex.spark.read.VortexScan: Logging-friendly readable description of the scan source.
dev.vortex.spark - package dev.vortex.spark
dev.vortex.spark.config - package dev.vortex.spark.config
dev.vortex.spark.read - package dev.vortex.spark.read
dev.vortex.spark.write - package dev.vortex.spark.write

E

endpoint() - Method in class dev.vortex.spark.config.VortexS3Properties
equals(Object) - Method in record class dev.vortex.spark.VortexFilePartition: Indicates whether some other object is "equal to" this one.
equals(Object) - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Indicates whether some other object is "equal to" this one.

F

filePath() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the path to the written Vortex file.
formatOptions() - Method in record class dev.vortex.spark.VortexFilePartition: Returns the value of the formatOptions record component.
fromArrowField(Field) - Static method in class dev.vortex.spark.ArrowUtils: Converts an Arrow Field to a Spark SQL DataType.
fromArrowType(ArrowType) - Static method in class dev.vortex.spark.ArrowUtils: Converts an Arrow type to a Spark SQL DataType.

G

get() - Method in interface dev.vortex.spark.VortexSessionProvider: Construct (or return a cached) Session.
get() - Static method in class dev.vortex.spark.VortexSparkSession: Returns the default JVM-wide session, creating it on first use.
get(Map<String, String>) - Static method in class dev.vortex.spark.VortexSparkSession: Resolve the session to use for a given set of Spark format options.
getArray(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the array value at the specified row.
getBinary(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the binary data (byte array) at the specified row.
getBoolean(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the boolean value at the specified row.
getByte(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the byte value at the specified row.
getChild(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the child column at the specified ordinal.
getDecimal(int, int, int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the decimal value at the specified row with the given precision and scale.
getDouble(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the double value at the specified row.
getFloat(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the float value at the specified row.
getInt(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the int value at the specified row.
getLong(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the long value at the specified row.
getMap(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the map value at the specified row.
getPartitionMessages() - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter.PartitionedWriterCommitMessage: Returns the commit messages from each individual partition writer.
getShort(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the short value at the specified row.
getTable(StructType, Transform[], Map<String, String>) - Method in class dev.vortex.spark.VortexDataSourceV2: Creates a Vortex table instance with the given schema and properties.
getUTF8String(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the UTF8String value at the specified row.
getValueVector() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the underlying Apache Arrow ValueVector wrapped by this column vector.

H

HadoopUtils - Class in dev.vortex.spark.config
hashCode() - Method in record class dev.vortex.spark.VortexFilePartition: Returns a hash code value for this object.
hashCode() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Returns a hash code value for this object.
hasNull() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns whether this column contains any null values.

I

inferPartitionColumnType(String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Infers a Spark DataType from a partition value string.
inferPartitioning(CaseInsensitiveStringMap) - Method in class dev.vortex.spark.VortexDataSourceV2: Infers partition transforms by inspecting Hive-style key=value segments in the first listed file path.
inferSchema(CaseInsensitiveStringMap) - Method in class dev.vortex.spark.VortexDataSourceV2: Infers the schema of the Vortex files specified in the options.
isNullAt(int) - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns whether the value at the specified row is null.

N

name() - Method in class dev.vortex.spark.VortexTable: Returns the name of this table.
newScanBuilder(CaseInsensitiveStringMap) - Method in class dev.vortex.spark.VortexTable: Creates a new ScanBuilder for this table.
newWriteBuilder(LogicalWriteInfo) - Method in class dev.vortex.spark.VortexTable: Creates a new WriteBuilder for writing data to this table.
numNulls() - Method in class dev.vortex.spark.read.VortexArrowColumnVector: Returns the total number of null values in this column.

O

onDataWriterCommit(WriterCommitMessage) - Method in class dev.vortex.spark.write.VortexBatchWrite: Called when a single data writer task completes successfully.

P

parsePartitionValues(String) - Static method in class dev.vortex.spark.read.PartitionPathUtils: Parses Hive-style key=value segments from a file path.
PartitionedVortexDataWriter - Class in dev.vortex.spark.write: Writes Spark InternalRow data to Vortex files organized in Hive-style partition directories.
PartitionedVortexDataWriter.PartitionedWriterCommitMessage - Class in dev.vortex.spark.write: Commit message that aggregates results from multiple partition writers.
partitioning() - Method in class dev.vortex.spark.VortexTable: Returns the partitioning transforms for this table.
PartitionPathUtils - Class in dev.vortex.spark.read: Utilities for discovering and materializing Hive-style partition columns from file paths.
partitionValues() - Method in record class dev.vortex.spark.VortexFilePartition: Returns the value of the partitionValues record component.
paths() - Method in record class dev.vortex.spark.VortexFilePartition: Returns the value of the paths record component.
planInputPartitions() - Method in class dev.vortex.spark.read.VortexBatchExec: Plans the input partitions for this batch scan.
PROVIDER_OPTION - Static variable in class dev.vortex.spark.VortexSparkSession: Options key used to select a VortexSessionProvider by class name.
pruneColumns(StructType) - Method in class dev.vortex.spark.read.VortexScanBuilder: Prunes the columns to only include those specified in the required schema.
pushedPredicates() - Method in class dev.vortex.spark.read.VortexScanBuilder: Returns the predicates this scan promises to apply.
pushPredicates(Predicate[]) - Method in class dev.vortex.spark.read.VortexScanBuilder: Splits the supplied predicates into pushed and not-pushed sets.

R

readSchema() - Method in class dev.vortex.spark.read.VortexScan: Returns the schema for the data that will be read by this scan.
readSchema() - Method in record class dev.vortex.spark.VortexFilePartition: Returns the value of the readSchema record component.
recordCount() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Gets the number of records written to the file.
region() - Method in class dev.vortex.spark.config.VortexS3Properties

S

s3PropertiesFromHadoopConf(Configuration) - Static method in class dev.vortex.spark.config.HadoopUtils
sasKey() - Method in class dev.vortex.spark.config.VortexAzureProperties
schema() - Method in class dev.vortex.spark.VortexTable: Returns the schema of this table.
secretAccessKey() - Method in class dev.vortex.spark.config.VortexS3Properties
sessionToken() - Method in class dev.vortex.spark.config.VortexS3Properties
setAccessKey(String) - Method in class dev.vortex.spark.config.VortexAzureProperties
setAccessKeyId(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setDefault(Session) - Static method in class dev.vortex.spark.VortexSparkSession: Replace the default session.
setEndpoint(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setRegion(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSasKey(String) - Method in class dev.vortex.spark.config.VortexAzureProperties
setSecretAccessKey(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSessionToken(String) - Method in class dev.vortex.spark.config.VortexS3Properties
setSkipSignature(boolean) - Method in class dev.vortex.spark.config.VortexAzureProperties
setSkipSignature(boolean) - Method in class dev.vortex.spark.config.VortexS3Properties
shortName() - Method in class dev.vortex.spark.VortexDataSourceV2: Returns the short name identifier for this data source.
skipSignature() - Method in class dev.vortex.spark.config.VortexAzureProperties
skipSignature() - Method in class dev.vortex.spark.config.VortexS3Properties
SparkToArrowSchema - Class in dev.vortex.spark.write: Utility class for converting Spark SQL schemas to Arrow schemas.
supportColumnarReads(InputPartition) - Method in class dev.vortex.spark.read.VortexPartitionReaderFactory
supportsExternalMetadata() - Method in class dev.vortex.spark.VortexDataSourceV2: Indicates whether this data source supports external metadata (schemas).

T

toBatch() - Method in class dev.vortex.spark.read.VortexScan: Converts this scan to a Batch for execution.
toBatch() - Method in class dev.vortex.spark.write.VortexBatchWrite: Returns this object as a BatchWrite.
toString() - Method in record class dev.vortex.spark.VortexFilePartition: Returns a string representation of this record class.
toString() - Method in record class dev.vortex.spark.write.VortexWriterCommitMessage: Returns a string representation of this record class.
truncate() - Method in class dev.vortex.spark.write.VortexWriteBuilder: Configures the write operation to truncate existing data.

V

VortexArrowColumnVector - Class in dev.vortex.spark.read: Spark ColumnVector implementation that wraps Apache Arrow vectors from Vortex data.
VortexArrowColumnVector(ValueVector) - Constructor for class dev.vortex.spark.read.VortexArrowColumnVector: Creates a new VortexArrowColumnVector wrapping the specified Arrow ValueVector.
VortexAzureProperties - Class in dev.vortex.spark.config
VortexAzureProperties() - Constructor for class dev.vortex.spark.config.VortexAzureProperties
VortexBatchExec - Class in dev.vortex.spark.read: Execution source for batch scans of Vortex file tables.
VortexBatchExec(List<String>, List<Column>, Map<String, String>, Predicate[]) - Constructor for class dev.vortex.spark.read.VortexBatchExec: Creates a new VortexBatchExec for scanning the specified Vortex files.
VortexBatchWrite - Class in dev.vortex.spark.write: Manages the batch write operation for creating Vortex files.
VortexDataSourceV2 - Class in dev.vortex.spark: Spark V2 data source for reading and writing Vortex files.
VortexDataSourceV2() - Constructor for class dev.vortex.spark.VortexDataSourceV2: Creates a new instance of the Vortex data source.
VortexDataWriter - Class in dev.vortex.spark.write: Writes Spark InternalRow data to a Vortex file.
VortexDataWriterFactory - Class in dev.vortex.spark.write: Factory for creating VortexDataWriter instances on Spark executors.
VortexFilePartition - Record Class in dev.vortex.spark: An InputPartition describing a group of Vortex files that a single reader should handle together.
VortexFilePartition(List<String>, StructType, Map<String, String>, Map<String, String>) - Constructor for record class dev.vortex.spark.VortexFilePartition: Creates an instance of a VortexFilePartition record class.
VortexPartitionReaderFactory - Class in dev.vortex.spark.read: Factory that produces columnar readers for Vortex files.
VortexPartitionReaderFactory(List<String>, Map<String, String>, Predicate[]) - Constructor for class dev.vortex.spark.read.VortexPartitionReaderFactory
VortexS3Properties - Class in dev.vortex.spark.config
VortexS3Properties() - Constructor for class dev.vortex.spark.config.VortexS3Properties
VortexScan - Class in dev.vortex.spark.read: Spark V2 Scan over a table of Vortex files.
VortexScan(List<String>, List<Column>, Map<String, String>, Predicate[]) - Constructor for class dev.vortex.spark.read.VortexScan: Creates a new VortexScan for the specified file paths and columns.
VortexScanBuilder - Class in dev.vortex.spark.read: Spark V2 ScanBuilder for table scans over Vortex files.
VortexScanBuilder(Map<String, String>) - Constructor for class dev.vortex.spark.read.VortexScanBuilder: Creates a new VortexScanBuilder with empty paths and columns.
VortexScanBuilder(Map<String, String>, Transform[]) - Constructor for class dev.vortex.spark.read.VortexScanBuilder: Creates a new VortexScanBuilder with empty paths and columns and the supplied partition transforms.
VortexSessionProvider - Interface in dev.vortex.spark: User hook for supplying a custom Session to Vortex Spark readers and writers.
VortexSparkSession - Class in dev.vortex.spark: JVM-wide holder for one or more Vortex Sessions used by Spark readers and writers.
VortexTable - Class in dev.vortex.spark: Spark V2 Table of Vortex files that supports both reading and writing.
VortexTable(ImmutableList<String>, StructType, Map<String, String>, Transform[]) - Constructor for class dev.vortex.spark.VortexTable: Creates a new VortexTable with read/write support.
VortexWriteBuilder - Class in dev.vortex.spark.write: Builder for configuring Vortex write operations.
VortexWriteBuilder(String, LogicalWriteInfo, Map<String, String>, Transform[]) - Constructor for class dev.vortex.spark.write.VortexWriteBuilder: Creates a new VortexWriteBuilder.
VortexWriterCommitMessage - Record Class in dev.vortex.spark.write: Commit message containing information about a successfully written Vortex file.
VortexWriterCommitMessage(String, long, long) - Constructor for record class dev.vortex.spark.write.VortexWriterCommitMessage: Creates a new commit message for a written Vortex file.

W

write(InternalRow) - Method in class dev.vortex.spark.write.PartitionedVortexDataWriter
write(InternalRow) - Method in class dev.vortex.spark.write.VortexDataWriter: Writes a single row to the Vortex file.

A B C D E F G H I N O P R S T V W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form