Package dev.vortex.spark

Class VortexTable

java.lang.Object
dev.vortex.spark.VortexTable
All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.SupportsRead, org.apache.spark.sql.connector.catalog.SupportsWrite, org.apache.spark.sql.connector.catalog.Table
public final class VortexTable extends Object implements org.apache.spark.sql.connector.catalog.Table, org.apache.spark.sql.connector.catalog.SupportsRead, org.apache.spark.sql.connector.catalog.SupportsWrite
Spark V2 Table of Vortex files that supports both reading and writing.

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexTable(com.google.common.collect.ImmutableList<String> paths, org.apache.spark.sql.types.StructType schema, Map<String,String> formatOptions)
    Creates a new VortexTable with read/write support.

  • Method Summary

    Modifier and Type
    Method
    Description
    Set<org.apache.spark.sql.connector.catalog.TableCapability>
    capabilities()
    Returns the capabilities supported by this table.
    String
    name()
    Returns the name of this table.
    org.apache.spark.sql.connector.read.ScanBuilder
    newScanBuilder(org.apache.spark.sql.util.CaseInsensitiveStringMap _options)
    Creates a new ScanBuilder for this table.
    org.apache.spark.sql.connector.write.WriteBuilder
    newWriteBuilder(org.apache.spark.sql.connector.write.LogicalWriteInfo info)
    Creates a new WriteBuilder for writing data to this table.
    org.apache.spark.sql.types.StructType
    schema()
    Returns the schema of this table.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.catalog.Table

    columns, partitioning, properties

  • Constructor Details

    • VortexTable

      public VortexTable(com.google.common.collect.ImmutableList<String> paths, org.apache.spark.sql.types.StructType schema, Map<String,String> formatOptions)
      Creates a new VortexTable with read/write support.

  • Method Details

    • newScanBuilder

      public org.apache.spark.sql.connector.read.ScanBuilder newScanBuilder(org.apache.spark.sql.util.CaseInsensitiveStringMap _options)
      Creates a new ScanBuilder for this table.

      The scan builder is pre-configured with all the file paths and columns from this table. The options parameter is currently unused but reserved for future use (e.g., S3 credentials).

      Specified by:
      newScanBuilder in interface org.apache.spark.sql.connector.catalog.SupportsRead
      Parameters:
      _options - scan options (currently unused)
      Returns:
      a new VortexScanBuilder configured for this table

    • name

      public String name()
      Returns the name of this table.

      The name includes the "vortex" prefix and a comma-separated list of all file paths that comprise this table.

      Specified by:
      name in interface org.apache.spark.sql.connector.catalog.Table
      Returns:
      the table name in the format: vortex."path1,path2,..."

    • schema

      public org.apache.spark.sql.types.StructType schema()
      Returns the schema of this table.

      The schema is derived from the columns available for reading, or from the explicit write schema if this table is being used for writing.

      Specified by:
      schema in interface org.apache.spark.sql.connector.catalog.Table
      Returns:
      the StructType representing the table schema

    • newWriteBuilder

      public org.apache.spark.sql.connector.write.WriteBuilder newWriteBuilder(org.apache.spark.sql.connector.write.LogicalWriteInfo info)
      Creates a new WriteBuilder for writing data to this table.

      The WriteBuilder is responsible for configuring and executing write operations to create new Vortex files.

      Specified by:
      newWriteBuilder in interface org.apache.spark.sql.connector.catalog.SupportsWrite
      Parameters:
      info - logical information about the write operation
      Returns:
      a new VortexWriteBuilder configured for this table

    • capabilities

      public Set<org.apache.spark.sql.connector.catalog.TableCapability> capabilities()
      Returns the capabilities supported by this table.

      Vortex tables support batch reading and batch writing.

      Specified by:
      capabilities in interface org.apache.spark.sql.connector.catalog.Table
      Returns:
      a set containing TableCapability.BATCH_READ and BATCH_WRITE