How to Remove header and footer from Dataframe?

Question

I am reading a text (not CSV) file that has header, content and footer using

spark.read.format(\"text\").option(\"delimiter\",\"|\")...load(file)

Answer 1

assuming the file is not so large we can use collect to get the dataframe as iterator and the access the last element as follows:

df = df.collect()[data.count()-1]

avoid using collect on large datasets.

or

we can use take to cut off the last row.

df = df.take(data.count()-1)