Python file.write creating extra carriage return

Question

I'm writing a series of SQL statements to a file using python. The template string looks like:

store_insert = '\tinsert stores (storenum, ...) values (\'%s\', ...)'

I'm writing to the file like so:

for line in source:
    line = line.rstrip()
    fields = line.split('\t')
    script.write(store_insert % tuple(fields))
    script.write(os.linesep)

However, in the resulting output, I see \r\r\n at the end of each line, rather than \r\n as I would expect. Why?

Answer 1

\n is converted to os.linesep for files opened in text-mode. So when you write os.linesep to a text-mode file on Windows, you write \r\n, and the \n gets converted resulting in \r\r\n.

See also the docs:

Do not use os.linesep as a line terminator when writing files opened in text mode (the default); use a single '\n' instead, on all platforms.

Answer 2

With Python 3

os.open() introduces the new parameter newline that allows to specify a string which any occurrence of \n will be translated to.

Passing an empty string argument newline='' disables the translation, leaving the new line char as it is. Valid for text mode only.

From the documentation

On output, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

Answer 3

Text files have different line endings on different operating systems, but it's convenient to work with strings that have a consistent line ending character. Python inherits the convention from C of using '\n' as the universal line ending character and relying on the file read and write functions to do a conversion, if necessary. The read and write functions know to do this if the file was opened in the default text mode. If you add the b character to the mode string when opening the file, this translation is skipped.

Answer 4

see the open() doc:

In addition to the standard fopen() values mode may be 'U' or 'rU'. Python is usually built with universal newline support; supplying 'U' opens the file as a text file, but lines may be terminated by any of the following: the Unix end-of-line convention '\n', the Macintosh convention '\r', or the Windows convention '\r\n'. All of these external representations are seen as '\n' by the Python program. If Python is built without universal newline support a mode with 'U' is the same as normal text mode. Note that file objects so opened also have an attribute called newlines which has a value of None (if no newlines have yet been seen), '\n', '\r', '\r\n', or a tuple containing all the newline types seen.

Answer 5

Works for me:

>>> import tempfile
>>> tmp = tempfile.TemporaryFile(mode="w+")
>>> store_insert = '\tinsert stores (storenum, ...) values (\'%s\', ...)'
>>> lines = ["foo\t\t"]
>>> for line in lines:
...     line = line.rstrip()
...     fields = line.split("\t")
...     tmp.write(store_insert % tuple(fields))
...     tmp.write(os.linesep)
...
>>> tmp.seek(0)
>>> tmp.read()
"\tinsert stores (storenum, ...) values ('foo', ...)\r\n"

Are you sure this is the code that's running, that os.linesep is what you think it is, etc?