Concatenate multiple files into a single file object without creating a new file

Question

This question is related to Python concatenate text files

I have a list of file_names, like [\'file1.txt\', \'file2.txt\', ...].

I wou

Answer 1

EDIT:

As pointed out in the comments, this solution probably would not be optimal for large files as it loads everything into memory. A solution using generators would be better if large files are involved. Thanks to LohmarASHAR for pointing that out!

You could just create on big list from all of the files by looping over the list of filenames. This wouldn't create a new file object, just a new list object:

filenames = ["f1.txt", "f2.txt", "f3.txt"]

# list to store each line of the file
output = []

# iterate over list of filenames
for text_file in filenames:

    # open file
    with open(text_file) as f:

        # iterate over each line in the file and add to output
        for line in f.readlines():
            output.append(line)

Not that we are not explicitly calling close() on the file, this is because the with... statement will close the file for us as soon as it goes out of scope.

If you are able to use external libraries, pandas might be worth looking into for storing the file data in efficient, easy-to-use objects.

Answer 2

Using built-ins:

product=[]
for File in ['file1.txt','file2.txt','file3.txt']:
    for line in open(File,'r').readlines():
        product.append(line)

for line in product:print(line)

file.readlines() outputs the contents to a list and the file is closed.

You could also write:

product=[]
for File in ['file1.txt','file2.txt','file3.txt']:
    product+=open(File).readlines()

It's shorter and probably faster but I use the first because it reads better to me.

Cheers

Answer 3

Let's say multiple_files is a list which contain all file names

multiple_files = ["file1.txt", "file2.txt", "file3.txt", ...] # and so on...

Open the output file which will contain all

f = open("multiple_files.txt", "w")
for _file in multiple_files:
    f.write(_file.read())

This way you don't have to read each and every line of your files.

Although the above method is simpler, You also have fileinput module as an alternative.

fileinput docs

You can use fileinput to access and process multiple files.

Example:

with fileinput.input(files=('file1.txt', 'file2.txt')) as f:
    for line in f:
        process(line)

Answer 4

try something along this lines:

def read_files(*filenames):
    for filename in filenames:
        with open(filename,'r') as file_obj:
            for line in file_obj:
                yield line

you can call it with

for line in read_files("f1.txt", "f2.txt", "f3.txt"):
    #... do whatever with the line

or

filenames = ["f1.txt", "f2.txt", "f3.txt"]
for line in read_files(*filenames):
    #... do whatever with the line

Answer 5

Instead of making python read multiple files, pipe the contents from the shell and read it from stdin. This will also make your program more flexible as you can pass in any set of files into your python program without changing your code.

Answer 6

Use input from fileinput module. It reads from multiple files but makes it look like the strings are coming from a single file. (Lazy line iteration).

import fileinput

files= ['F:/files/a.txt','F:/files/c.txt','F:/files/c.txt']

allfiles = fileinput.input(files)

for line in allfiles: # this will iterate over lines in all the files
    print(line)

# or read lines like this: allfiles.readline()

If you need all the text in one place use StringIO

import io

files= ['F:/files/a.txt','F:/files/c.txt','F:/files/c.txt']


lines = io.StringIO()   #file like object to store all lines

for file_dir in files:
    with open(file_dir, 'r') as file:
        lines.write(file.read())
        lines.write('\n')

lines.seek(0)        # now you can treat this like a file like object
print(lines.read())