Package dev.vortex.spark

Record Class VortexFilePartition

java.lang.Object
java.lang.Record
dev.vortex.spark.VortexFilePartition
Record Components:
paths - the Vortex file paths (or globs) belonging to this input partition
readSchema - the requested output schema (data columns + partition columns)
formatOptions - object-store properties used to open the files
partitionValues - Hive-style partition column values shared by all paths()
All Implemented Interfaces:
Serializable, org.apache.spark.sql.connector.read.InputPartition
public record VortexFilePartition(List<String> paths, org.apache.spark.sql.types.StructType readSchema, Map<String,String> formatOptions, Map<String,String> partitionValues) extends Record implements org.apache.spark.sql.connector.read.InputPartition, Serializable
An InputPartition describing a group of Vortex files that a single reader should handle together.

Each executor opens a single Vortex Session, DataSource and Scan over the partition's paths() and consumes every Vortex partition produced by that scan before moving on to the next Spark InputPartition.

The requested output schema is carried as a StructType rather than a list of Column objects: StructType is the stable serialization surface in Spark and survives shipping to executors reliably.

See Also:

  • Constructor Details

    • VortexFilePartition

      public VortexFilePartition(List<String> paths, org.apache.spark.sql.types.StructType readSchema, Map<String,String> formatOptions, Map<String,String> partitionValues)
      Creates an instance of a VortexFilePartition record class.
      Parameters:
      paths - the value for the paths record component
      readSchema - the value for the readSchema record component
      formatOptions - the value for the formatOptions record component
      partitionValues - the value for the partitionValues record component

  • Method Details

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. All components in this record class are compared with Objects::equals(Object,Object).
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • paths

      public List<String> paths()
      Returns the value of the paths record component.
      Returns:
      the value of the paths record component

    • readSchema

      public org.apache.spark.sql.types.StructType readSchema()
      Returns the value of the readSchema record component.
      Returns:
      the value of the readSchema record component

    • formatOptions

      public Map<String,String> formatOptions()
      Returns the value of the formatOptions record component.
      Returns:
      the value of the formatOptions record component

    • partitionValues

      public Map<String,String> partitionValues()
      Returns the value of the partitionValues record component.
      Returns:
      the value of the partitionValues record component