How to load data into pandas from a large database?

Question

I have a postgres database which contains time series data.The size of the database is around 1 GB.Currently to read data, this is what I do

import psycopg2
         


        

Answer 1

Consider loading this file directly to the database using psql. From your console try this command:

$ cat file.csv | psql db -c "COPY your_table FROM STDIN DELIMITER ',' CSV HEADER"

Example:

Consider the following table structure ..

CREATE TEMPORARY TABLE t (foo INT, bar TEXT);

and the following CSV file ..

x,y
1,foo
2,bar
3,foo-bar

Execute the following command from your terminal:

$ cat /tmp/file.csv | psql db -c "COPY t FROM STDIN DELIMITER ',' CSV HEADER"

And here is your data:

db=# SELECT * FROM t;
 a |    b    
---+---------
 1 | foo
 2 | bar
 3 | foo-bar
(3 Zeilen)

Answer 2

pd.read_sql() also has parameter chunksize, so you can read data from SQL table/query in chunks:

for df in pd.read_sql("Select * from timeseries", conn, chunksize=10**4):
    # process `df` chunk here...