org.apache.spark.sql.json

Class JsonRDD

Object

org.apache.spark.sql.json.JsonRDD

All Implemented Interfaces:

Logging

public class JsonRDD
extends Object
implements Logging

Constructor Summary

Constructors

Constructor and Description
JsonRDD()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.spark.sql.types.DataType	compatibleType(org.apache.spark.sql.types.DataType t1, org.apache.spark.sql.types.DataType t2) Returns the most general data type for two given data types.
static Object	enforceCorrectType(Object value, org.apache.spark.sql.types.DataType desiredType)
static org.apache.spark.sql.types.StructType	inferSchema(RDD<String> json, double samplingRatio, String columnNameOfCorruptRecords)
static RDD<org.apache.spark.sql.Row>	jsonStringToRow(RDD<String> json, org.apache.spark.sql.types.StructType schema, String columnNameOfCorruptRecords)
static org.apache.spark.sql.types.StructType	nullTypeToStringType(org.apache.spark.sql.types.StructType struct)
static void	rowToJSON(org.apache.spark.sql.types.StructType rowSchema, com.fasterxml.jackson.core.JsonGenerator gen, org.apache.spark.sql.Row row) Transforms a single Row to JSON using Jackson

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JsonRDD

public JsonRDD()

Method Detail

jsonStringToRow

public static RDD<org.apache.spark.sql.Row> jsonStringToRow(RDD<String> json,
                                            org.apache.spark.sql.types.StructType schema,
                                            String columnNameOfCorruptRecords)

inferSchema

public static org.apache.spark.sql.types.StructType inferSchema(RDD<String> json,
                                                double samplingRatio,
                                                String columnNameOfCorruptRecords)

nullTypeToStringType

public static org.apache.spark.sql.types.StructType nullTypeToStringType(org.apache.spark.sql.types.StructType struct)

compatibleType

public static org.apache.spark.sql.types.DataType compatibleType(org.apache.spark.sql.types.DataType t1,
                                                 org.apache.spark.sql.types.DataType t2)

Returns the most general data type for two given data types.

enforceCorrectType

public static Object enforceCorrectType(Object value,
                        org.apache.spark.sql.types.DataType desiredType)

rowToJSON

public static void rowToJSON(org.apache.spark.sql.types.StructType rowSchema,
             com.fasterxml.jackson.core.JsonGenerator gen,
             org.apache.spark.sql.Row row)

Transforms a single Row to JSON using Jackson

Parameters:

rowSchema - the schema object used for conversion

gen - a JsonGenerator object

row - The row to convert