Convert Spark DataFrame to Pojo Object

Question

Please see below code:

    //Create Spark Context
    SparkConf sparkConf = new SparkConf().setAppName(\"TestWithObjects\").setMaster(\"local\");
    JavaSpa         


        

Answer 1

A DataFrame is stored as Rows, so you can use the methods there to cast from untyped to typed. Take a look at the get methods.

Answer 2

DataFrame is simply a type alias of Dataset[Row] . These operations are also referred as “untyped transformations” in contrast to “typed transformations” that come with strongly typed Scala/Java Datasets.

The conversion from Dataset[Row] to Dataset[Person] is very simple in spark

DataFrame result = sQLContext.sql("SELECT * FROM peoples WHERE name='test'");

At this point, Spark converts your data into DataFrame = Dataset[Row], a collection of generic Row object, since it does not know the exact type.

// Create an Encoders for Java beans
Encoder<Person> personEncoder = Encoders.bean(Person.class); 
Dataset<Person> personDF = result.as(personEncoder);
personDF.show();

Now, Spark converts the Dataset[Row] -> Dataset[Person] type-specific Scala / Java JVM object, as dictated by the class Person.

Please refer to below link provided by databricks for further details

https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html