Retrieving Data from MySQL in batches via Python

Question

I would like to make this process in batches, because of the volume.

Here\'s my code:

 getconn = conexiones()
 con = getconn.mysqlDWconnect()
 with con:         


        

Answer 1

To expand on akalikin's answer, you can use a stepped iteration to split the query into chunks, and then use LIMIT and OFFSET to execute the query.

cur = con.cursor(mdb.cursors.DictCursor)
cur.execute("SELECT COUNT(*) FROM sales")

for i in range(0,cur.fetchall(),5):
    cur2 = con.cursor(mdb.cursors.DictCursor)
    cur2.execute("SELECT id, date, product_id, sales FROM sales LIMIT %s OFFSET %s" %(5,i))
    rows = cur2.fetchall()
    print rows