How to Remove header and footer from Dataframe?

Question

I am reading a text (not CSV) file that has header, content and footer using

spark.read.format(\"text\").option(\"delimiter\",\"|\")...load(file)

Answer 1

Assuming your text file has JSON header and Footer, Spark SQL way,

Sample Data

{"":[{:},{:}]}

Here the header can be avoided by following 3 lines (Assumption No Tilda in data),

jsonToCsvDF=spark.read.format("com.databricks.spark.csv").option("delimiter", "~").load()

jsonToCsvDF.createOrReplaceTempView("json_to_csv")

spark.sql("SELECT SUBSTR(`_c0`,5,length(`_c0`)-5) FROM json_to_csv").coalesce(1).write.option("header",false).mode("overwrite").text()

Now the output will look like,

[{:},{:}]

Hope it helps.