Package dev.vortex.spark

Class VortexFilePartition

java.lang.Object
dev.vortex.spark.VortexFilePartition
All Implemented Interfaces:
Serializable, org.apache.spark.sql.connector.read.InputPartition
public final class VortexFilePartition extends Object implements org.apache.spark.sql.connector.read.InputPartition, Serializable
An InputPartition for reading a whole Vortex file.

This class represents a partition that corresponds to a single Vortex file. It contains the file path and the columns to be read from that file. Each partition can be processed independently by Spark executors.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexFilePartition(String path, com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column> columns, com.google.common.collect.ImmutableMap<String,String> formatOptions, com.google.common.collect.ImmutableMap<String,String> partitionValues)
    Creates a new Vortex file partition.

  • Method Summary

    Modifier and Type
    Method
    Description
    com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column>
    getColumns()
    Returns the list of columns to be read from this partition.
    Map<String,String>
    getFormatOptions()
     
    com.google.common.collect.ImmutableMap<String,String>
    getPartitionValues()
    Returns the partition column values parsed from this file's Hive-style directory path.
    String
    getPath()
    Returns the file system path to the Vortex file for this partition.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.read.InputPartition

    preferredLocations

  • Constructor Details

    • VortexFilePartition

      public VortexFilePartition(String path, com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column> columns, com.google.common.collect.ImmutableMap<String,String> formatOptions, com.google.common.collect.ImmutableMap<String,String> partitionValues)
      Creates a new Vortex file partition.
      Parameters:
      path - the file system path to the Vortex file
      columns - the list of columns to read from the file
      formatOptions - options for accessing the file (S3/Azure credentials, etc.)
      partitionValues - Hive-style partition column values extracted from the file path

  • Method Details

    • getPath

      public String getPath()
      Returns the file system path to the Vortex file for this partition.
      Returns:
      the file path

    • getColumns

      public com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column> getColumns()
      Returns the list of columns to be read from this partition.
      Returns:
      the immutable list of columns

    • getFormatOptions

      public Map<String,String> getFormatOptions()

    • getPartitionValues

      public com.google.common.collect.ImmutableMap<String,String> getPartitionValues()
      Returns the partition column values parsed from this file's Hive-style directory path. Keys are column names, values are the string-encoded partition values.
      Returns:
      the partition values, empty if the file is not in a partitioned directory