Package dev.vortex.spark.write

Class PartitionedVortexDataWriter

java.lang.Object

dev.vortex.spark.write.PartitionedVortexDataWriter

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>

public final class PartitionedVortexDataWriter
extends Object
implements org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>, AutoCloseable

Writes Spark InternalRow data to Vortex files organized in Hive-style partition directories.

Supports the standard Spark partition transforms: identity, years, months, days, hours, and bucket. For each unique combination of evaluated transform values, a separate subdirectory is created and a dedicated VortexDataWriter writes data within it.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	PartitionedVortexDataWriter.PartitionedWriterCommitMessage	Commit message that aggregates results from multiple partition writers.

Method Summary

Modifier and Type	Method	Description
void	abort()
void	close()
org.apache.spark.sql.connector.write.WriterCommitMessage	commit()
void	write(org.apache.spark.sql.catalyst.InternalRow row)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.connector.write.DataWriter

currentMetricsValues, write, writeAll

Method Details

write

public void write(org.apache.spark.sql.catalyst.InternalRow row)
           throws IOException

Specified by:

write in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>

Throws:

IOException

commit

public org.apache.spark.sql.connector.write.WriterCommitMessage commit()
                                                                throws IOException

Specified by:

commit in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>

Throws:

IOException

abort

public void abort()
           throws IOException

Specified by:

abort in interface org.apache.spark.sql.connector.write.DataWriter<org.apache.spark.sql.catalyst.InternalRow>

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException