piping postgres COPY in python with psycopg2

Question

I\'m writing a script to do a copy of some data between two machines on the same network using psycopg2. I\'m replacing some old, ugly bash that does the copy with

Answer 1

You could use a deque that you've subclassed to support reading and writing:

from collections import deque
from Exceptions import IndexError

class DequeBuffer(deque):
    def write(self, data):
        self.append(data)
    def read(self):
        try:
            return self.popleft()
        except IndexError:
            return ''

buf = DequeBuffer()

If the reader is much faster than the writer, and the table is large, the deque will still get big, but it will be smaller than storing the whole thing.

Also, I don't know for sure return '' when the deque is empty is safe, rather than retrying until it's not empty, but I'd guess it is. Let me know if it works.

Remember to del buf when you're sure the copy is done, especially if the script isn't just exiting at that point.

Answer 2

You will have to put one of your calls in a separate thread. I just realized you can use os.pipe(), which makes the rest quite straightforward:

#!/usr/bin/python
import psycopg2
import os
import threading

fromdb = psycopg2.connect("dbname=from_db")
todb = psycopg2.connect("dbname=to_db")

r_fd, w_fd = os.pipe()

def copy_from():
    cur = todb.cursor()
    cur.copy_from(os.fdopen(r_fd), 'table')
    cur.close()
    todb.commit()

to_thread = threading.Thread(target=copy_from)
to_thread.start()

cur = fromdb.cursor()
write_f = os.fdopen(w_fd, 'w')
cur.copy_to(write_f, 'table')
write_f.close()   # or deadlock...

to_thread.join()