Appending two CSV files column-wise

Question

Suppose I have two CSV files called A and B in Python.

A\'s head looks like:

 headerName         


        

Answer 1

You might be able to get away with a named pipe. You have a Python process run which creates a pipe and opens it in write mode. It then outputs to that the column wise concatenation of the CSV files (similar to what you've got) already... When another process starts reading that file, it'll be able to consume the data, but no file is actually stored on the server, it's just on demand. When the "file" is consumed, then there'll be nothing in it, and any attempt to access it will block until another process writes to the other end.

Some dummy code - will need more thought out exception handling etc...:

import os
from itertools import izip

a = 'abcdef' # File A's rows
b = 'ghijkl' # File B's rows

outname = 'joined'

try:
    os.unlink(outname)
    os.mkfifo(outname)
except OSError:
    pass

with open(outname, 'w') as fout:
    for items in izip(a, b):
        fout.write(''.join(items) + '\n') # Do "real" write here instead...
    os.unlink(outname)

Something else opens that "file" in read mode and consumes it to retrieve the data. This should work unless that process has to have "physical files"...

Answer 2

If you get two file handles for the same file - one in 'read' mode, one in 'update' mode (r+b), the same strategy should work.

from itertools import izip
import csv
with open('A','rb') as f1, open('B','rb') as f2, open('A','r+b') as w:
    writer = csv.writer(w)
    for r1,r2 in izip(csv.reader(f1),csv.reader(f2)):
        writer.writerow(r1+r2)

When possible I'd recommend against this kind of thing and just explicitly write to a third file.