Package dev.vortex.spark.write

Class VortexDataWriter

java.lang.Object
dev.vortex.spark.write.VortexDataWriter
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>
public final class VortexDataWriter extends Object implements org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>, AutoCloseable
Writes Spark InternalRow data to a Vortex file.

This writer converts Spark's internal row format to Arrow vectors and writes them to a Vortex file using the Vortex writer API.

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexDataWriter(String filePath, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
    Creates a new VortexDataWriter.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    abort()
    Aborts the write operation and cleans up resources.
    void
    close()
    Closes the writer and releases resources.
    org.apache.spark.sql.connector.write.WriterCommitMessage
    commit()
    Commits the write operation and returns a commit message.
    void
    write(org.apache.spark.sql.catalyst.InternalRow row)
    Writes a single row to the Vortex file.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.write.DataWriter

    currentMetricsValues

  • Constructor Details

    • VortexDataWriter

      public VortexDataWriter(String filePath, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Creates a new VortexDataWriter.
      Parameters:
      filePath - the path where the Vortex file will be written
      schema - the schema of the data to write
      options - additional write options

  • Method Details

    • write

      public void write(org.apache.spark.sql.catalyst.InternalRow row) throws IOException
      Writes a single row to the Vortex file.

      Rows are batched and converted to Arrow format before writing.

      Specified by:
      write in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>
      Parameters:
      row - the row to write
      Throws:
      IOException - if writing fails

    • commit

      public org.apache.spark.sql.connector.write.WriterCommitMessage commit() throws IOException
      Commits the write operation and returns a commit message.

      This flushes any remaining rows and closes the Vortex writer.

      Specified by:
      commit in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>
      Returns:
      a commit message with file information
      Throws:
      IOException - if commit fails

    • abort

      public void abort() throws IOException
      Aborts the write operation and cleans up resources.

      This deletes any partially written file.

      Specified by:
      abort in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>
      Throws:
      IOException - if abort fails

    • close

      public void close() throws IOException
      Closes the writer and releases resources.

      This method ensures resources are cleaned up even if commit() or abort() were not called, making the class safe for use with try-with-resources.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException