Package dev.vortex.spark.write

Class SparkToArrowSchema

java.lang.Object
dev.vortex.spark.write.SparkToArrowSchema
public final class SparkToArrowSchema extends Object
Utility class for converting Spark SQL schemas to Arrow schemas. This enables the conversion of Spark DataFrames to Arrow format for writing to Vortex files.

  • Method Summary

    Modifier and Type
    Method
    Description
    static dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Schema
    convert(org.apache.spark.sql.types.StructType sparkSchema)
    Converts a Spark StructType schema to an Arrow Schema.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • convert

      public static dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Schema convert(org.apache.spark.sql.types.StructType sparkSchema)
      Converts a Spark StructType schema to an Arrow Schema.
      Parameters:
      sparkSchema - the Spark schema to convert
      Returns:
      the corresponding Arrow schema