How to load data into pandas from a large database?

Question

I have a postgres database which contains time series data.The size of the database is around 1 GB.Currently to read data, this is what I do

import psycopg2
         


        

Answer 1

Consider loading this file directly to the database using psql. From your console try this command:

$ cat file.csv | psql db -c "COPY your_table FROM STDIN DELIMITER ',' CSV HEADER"

Example:

Consider the following table structure ..

CREATE TEMPORARY TABLE t (foo INT, bar TEXT);

and the following CSV file ..

x,y
1,foo
2,bar
3,foo-bar

Execute the following command from your terminal:

$ cat /tmp/file.csv | psql db -c "COPY t FROM STDIN DELIMITER ',' CSV HEADER"

And here is your data:

db=# SELECT * FROM t;
 a |    b    
---+---------
 1 | foo
 2 | bar
 3 | foo-bar
(3 Zeilen)