Python writelines() and write() huge time difference

Question

I was working on a script which reading a folder of files(each of size ranging from 20 MB to 100 MB), modifies some data in each line, and writes back to a copy of the file.

Answer 1

as a complement to Martijn answer, the best way would be to avoid to build the list using join in the first place

Just pass a generator comprehension to writelines, adding the newline in the end: no unnecessary memory allocation and no loop (besides the comprehension)

myWrite.writelines("{}\n".format(x) for x in my_list)