Python and Django OperationalError (2006, 'MySQL server has gone away')

Question

Original: I have recently started getting MySQL OperationalErrors from some of my old code and cannot seem to trace back the problem. Since it was working before, I thought

Answer 1

How old is this code? Django has had databases defined in settings since at least .96. Only other thing I can think of is multi-db support, which changed things a bit, but even that was 1.1 or 1.2.

Even if you need a special DB for certain views, I think you'd probably be better off defining it in settings.

Answer 2

This error is mysterious because MySQL doesn't report why it disconnects, it just goes away.

It seems there are many causes of this kind of disconnection. One I just found is, if the query string too large, the server will disconnect. This probably relates to the max_allowed_packets setting.

Answer 3

This might be due to DB connections getting copied in your child threads from the main thread. I faced the same error when using python's multiprocessing library to spawn different processes. The connection objects are copied between processes during forking and it leads to MySQL OperationalErrors when making DB calls in the child thread.

Here's a good reference to solve this: Django multiprocessing and database connections

Answer 4

SQLAlchemy now has a great write-up on how you can use pinging to be pessimistic about your connection's freshness:

http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic

From there,

from sqlalchemy import exc
from sqlalchemy import event
from sqlalchemy.pool import Pool

@event.listens_for(Pool, "checkout")
def ping_connection(dbapi_connection, connection_record, connection_proxy):
    cursor = dbapi_connection.cursor()
    try:
        cursor.execute("SELECT 1")
    except:
        # optional - dispose the whole pool
        # instead of invalidating one at a time
        # connection_proxy._pool.dispose()

        # raise DisconnectionError - pool will try
        # connecting again up to three times before raising.
        raise exc.DisconnectionError()
    cursor.close()

And a test to make sure the above works:

from sqlalchemy import create_engine
e = create_engine("mysql://scott:tiger@localhost/test", echo_pool=True)
c1 = e.connect()
c2 = e.connect()
c3 = e.connect()
c1.close()
c2.close()
c3.close()

# pool size is now three.

print "Restart the server"
raw_input()

for i in xrange(10):
    c = e.connect()
    print c.execute("select 1").fetchall()
    c.close()

Answer 5

As per the MySQL documentation, your error message is raised when the client can't send a question to the server, most likely because the server itself has closed the connection. In the most common case the server will close an idle connection after a (default) of 8 hours. This is configurable on the server side.

The MySQL documentation gives a number of other possible causes which might be worth looking into to see if they fit your situation.

An alternative to calling connect() in every function (which might end up needlessly creating new connections) would be to investigate using the ping() method on the connection object; this tests the connection with the option of attempting an automatic reconnect. I struggled to find some decent documentation for the ping() method online, but the answer to this question might help.

Note, automatically reconnecting can be dangerous when handling transactions as it appears the reconnect causes an implicit rollback (and appears to be the main reason why autoreconnect is not a feature of the MySQLdb implementation).

Answer 6

Check if you are allowed to create mysql connection object in one thread and then use it in another.

If it's forbidden, use threading.Local for per-thread connections:

class Db(threading.local):
    """ thread-local db object """
    con = None

    def __init__(self, ...options...):
        super(Db, self).__init__()
        self.con = MySQLdb.connect(...options...)

db1 = Db(...)


def test():
    """safe to run from any thread"""
    cursor = db.con.cursor()
    cursor.execute(...)