Read File spark, set field having specific value to null or “”

Question

I\'m Reading a text file delimited with | . There are some fields having value \\N . When read the file row by row to a data-frame, is there any way to make the field having

Answer 1

Once you load the dataframe use the when condition on all the columns in generic way

inputDf.select(inputDf.columns.map(c=> when(col(c) === “\\N”,””).otherwise(col(c)).alias(c)):_*).show