Package dev.vortex.spark.read

Class VortexScan

java.lang.Object
dev.vortex.spark.read.VortexScan
All Implemented Interfaces:
org.apache.spark.sql.connector.read.Scan
public final class VortexScan extends Object implements org.apache.spark.sql.connector.read.Scan
Spark V2 Scan over a table of Vortex files.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.sql.connector.read.Scan

    org.apache.spark.sql.connector.read.Scan.ColumnarSupportMode

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexScan(com.google.common.collect.ImmutableList<String> paths, com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column> readColumns, Map<String,String> formatOptions)
    Creates a new VortexScan for the specified file paths and columns.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.read.Scan.ColumnarSupportMode
    columnarSupportMode()
    Returns the columnar support mode for this scan.
    String
    description()
    Logging-friendly readable description of the scan source.
    org.apache.spark.sql.types.StructType
    readSchema()
    Returns the schema for the data that will be read by this scan.
    org.apache.spark.sql.connector.read.Batch
    toBatch()
    Converts this scan to a Batch for execution.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.read.Scan

    reportDriverMetrics, supportedCustomMetrics, toContinuousStream, toMicroBatchStream

  • Constructor Details

    • VortexScan

      public VortexScan(com.google.common.collect.ImmutableList<String> paths, com.google.common.collect.ImmutableList<org.apache.spark.sql.connector.catalog.Column> readColumns, Map<String,String> formatOptions)
      Creates a new VortexScan for the specified file paths and columns.
      Parameters:
      paths - the list of Vortex file paths to scan
      readColumns - the list of columns to read from the files

  • Method Details

    • readSchema

      public org.apache.spark.sql.types.StructType readSchema()
      Returns the schema for the data that will be read by this scan.

      The schema is constructed from the read columns that were specified when this scan was created.

      Specified by:
      readSchema in interface org.apache.spark.sql.connector.read.Scan
      Returns:
      the StructType representing the schema of the read data

    • description

      public String description()
      Logging-friendly readable description of the scan source.
      Specified by:
      description in interface org.apache.spark.sql.connector.read.Scan

    • toBatch

      public org.apache.spark.sql.connector.read.Batch toBatch()
      Converts this scan to a Batch for execution.

      Creates a VortexBatchExec that will handle the actual reading of the specified files and columns.

      Specified by:
      toBatch in interface org.apache.spark.sql.connector.read.Scan
      Returns:
      a Batch implementation for executing this scan

    • columnarSupportMode

      public org.apache.spark.sql.connector.read.Scan.ColumnarSupportMode columnarSupportMode()
      Returns the columnar support mode for this scan.

      Vortex always provides columnar data access, so this method always returns SUPPORTED.

      Specified by:
      columnarSupportMode in interface org.apache.spark.sql.connector.read.Scan
      Returns:
      ColumnarSupportMode.SUPPORTED