How to avoid creating duplicates during data ingestion via spark?

Question

I have to read and store newly arriving files each day. The files are simple csv files, but the problem is that sometimes the newly arriving logfile contains the whole previ