How to modify a text file?

Question

I\'m using Python, and would like to insert a string into a text file without deleting or copying the file. How can I do that?

Answer 1

The fileinput module of the Python standard library will rewrite a file inplace if you use the inplace=1 parameter:

import sys
import fileinput

# replace all occurrences of 'sit' with 'SIT' and insert a line after the 5th
for i, line in enumerate(fileinput.input('lorem_ipsum.txt', inplace=1)):
    sys.stdout.write(line.replace('sit', 'SIT'))  # replace 'sit' and write
    if i == 4: sys.stdout.write('\n')  # write a blank line after the 5th line

Answer 2

Rewriting a file in place is often done by saving the old copy with a modified name. Unix folks add a ~ to mark the old one. Windows folks do all kinds of things -- add .bak or .old -- or rename the file entirely or put the ~ on the front of the name.

import shutil
shutil.move( afile, afile+"~" )

destination= open( aFile, "w" )
source= open( aFile+"~", "r" )
for line in source:
    destination.write( line )
    if <some condition>:
        destination.write( >some additional line> + "\n" )
source.close()
destination.close()

Instead of shutil, you can use the following.

import os
os.rename( aFile, aFile+"~" )

Answer 3

If you know some unix you could try the following:

Notes: $ means the command prompt

Say you have a file my_data.txt with content as such:

$ cat my_data.txt
This is a data file
with all of my data in it.

Then using the os module you can use the usual sed commands

import os

# Identifiers used are:
my_data_file = "my_data.txt"
command = "sed -i 's/all/none/' my_data.txt"

# Execute the command
os.system(command)

If you aren't aware of sed, check it out, it is extremely useful.

Answer 4

Wrote a small class for doing this cleanly.

import tempfile

class FileModifierError(Exception):
    pass

class FileModifier(object):

    def __init__(self, fname):
        self.__write_dict = {}
        self.__filename = fname
        self.__tempfile = tempfile.TemporaryFile()
        with open(fname, 'rb') as fp:
            for line in fp:
                self.__tempfile.write(line)
        self.__tempfile.seek(0)

    def write(self, s, line_number = 'END'):
        if line_number != 'END' and not isinstance(line_number, (int, float)):
            raise FileModifierError("Line number %s is not a valid number" % line_number)
        try:
            self.__write_dict[line_number].append(s)
        except KeyError:
            self.__write_dict[line_number] = [s]

    def writeline(self, s, line_number = 'END'):
        self.write('%s\n' % s, line_number)

    def writelines(self, s, line_number = 'END'):
        for ln in s:
            self.writeline(s, line_number)

    def __popline(self, index, fp):
        try:
            ilines = self.__write_dict.pop(index)
            for line in ilines:
                fp.write(line)
        except KeyError:
            pass

    def close(self):
        self.__exit__(None, None, None)

    def __enter__(self):
        return self

    def __exit__(self, type, value, traceback):
        with open(self.__filename,'w') as fp:
            for index, line in enumerate(self.__tempfile.readlines()):
                self.__popline(index, fp)
                fp.write(line)
            for index in sorted(self.__write_dict):
                for line in self.__write_dict[index]:
                    fp.write(line)
        self.__tempfile.close()

Then you can use it this way:

with FileModifier(filename) as fp:
    fp.writeline("String 1", 0)
    fp.writeline("String 2", 20)
    fp.writeline("String 3")  # To write at the end of the file

Answer 5

Unfortunately there is no way to insert into the middle of a file without re-writing it. As previous posters have indicated, you can append to a file or overwrite part of it using seek but if you want to add stuff at the beginning or the middle, you'll have to rewrite it.

This is an operating system thing, not a Python thing. It is the same in all languages.

What I usually do is read from the file, make the modifications and write it out to a new file called myfile.txt.tmp or something like that. This is better than reading the whole file into memory because the file may be too large for that. Once the temporary file is completed, I rename it the same as the original file.

This is a good, safe way to do it because if the file write crashes or aborts for any reason, you still have your untouched original file.

Answer 6

As mentioned by Adam you have to take your system limitations into consideration before you can decide on approach whether you have enough memory to read it all into memory replace parts of it and re-write it.

If you're dealing with a small file or have no memory issues this might help:

Option 1) Read entire file into memory, do a regex substitution on the entire or part of the line and replace it with that line plus the extra line. You will need to make sure that the 'middle line' is unique in the file or if you have timestamps on each line this should be pretty reliable.

# open file with r+b (allow write and binary mode)
f = open("file.log", 'r+b')   
# read entire content of file into memory
f_content = f.read()
# basically match middle line and replace it with itself and the extra line
f_content = re.sub(r'(middle line)', r'\1\nnew line', f_content)
# return pointer to top of file so we can re-write the content with replaced string
f.seek(0)
# clear file content 
f.truncate()
# re-write the content with the updated content
f.write(f_content)
# close file
f.close()

Option 2) Figure out middle line, and replace it with that line plus the extra line.

# open file with r+b (allow write and binary mode)
f = open("file.log" , 'r+b')   
# get array of lines
f_content = f.readlines()
# get middle line
middle_line = len(f_content)/2
# overwrite middle line
f_content[middle_line] += "\nnew line"
# return pointer to top of file so we can re-write the content with replaced string
f.seek(0)
# clear file content 
f.truncate()
# re-write the content with the updated content
f.write(''.join(f_content))
# close file
f.close()