Defining DataFrame Schema for a table with 1500 columns in Spark

Question

I have a table with around 1500 columns in SQL Server. I need to read the data from this table and then convert it to proper datatype format and then insert the records into Ora

Answer 1

The options for reading a table with 1500 columns

1) Using Case class

Case class would not work because its limited to 22 fields( for scala version < 2.11).

2) Using StructType

You can use the StructType to define the schema and create the dataframe.

Third option

You can use the Spark-csv package . In this, you can use .option("inferschema","true"). This will automatically read the schema from the file.