Pandas SQL chunksize

Question

This is more of a question on understanding than programming. I am quite new to Pandas and SQL. I am using pandas to read data from SQL with some specific chunksize. When I run

Answer 1

Let's consider two options and what happens in both cases:

1. chunksize is None(default value):
   • pandas passes query to database
   • database executes query
   • pandas checks and sees that chunksize is None
   • pandas tells database that it wants to receive all rows of the result table at once
   • database returns all rows of the result table
   • pandas stores the result table in memory and wraps it into a data frame
   • now you can use the data frame
2. chunksize in not None:
   • pandas passes query to database
   • database executes query
   • pandas checks and sees that chunksize has some value
   • pandas creates a query iterator(usual 'while True' loop which breaks when database says that there is no more data left) and iterates over it each time you want the next chunk of the result table
   • pandas tells database that it wants to receive chunksize rows
   • database returns the next chunksize rows from the result table
   • pandas stores the next chunksize rows in memory and wraps it into a data frame
   • now you can use the data frame

For more details you can see pandas\io\sql.py module, it is well documented