What is the proper way to handle Redis connection in Tornado ? (Async - Pub/Sub)

Question

I am using Redis along with my Tornado application with asyc client Brukva, when I looked at the sample apps at Brukva site they are making new connection on \"init

Answer 1

you should pool the connections in your app. since it seems like brukva doesn't support this automatically (redis-py supports this, but is blocking by nature so it doesn't go well with tornado), you need to write your own connection pool.

the pattern is pretty simple, though. something along these lines (this is not real operational code):

class BrukvaPool():

    __conns = {}


    def get(host, port,db):
        ''' Get a client for host, port, db '''

        key = "%s:%s:%s" % (host, port, db)

        conns = self.__conns.get(key, [])
        if conns:
            ret = conns.pop()
            return ret
        else:
           ## Init brukva client here and connect it

    def release(client):
        ''' release a client at the end of a request '''
        key = "%s:%s:%s" % (client.connection.host, client.connection.port, client.connection.db)
        self.__conns.setdefault(key, []).append(client)

it can be a bit more tricky, but that's the main idea.

Answer 2

A little late but, I've been using tornado-redis. It works with tornado's ioloop and the tornado.gen module

Install tornadoredis

It can be installed from pip

pip install tornadoredis

or with setuptools

easy_install tornadoredis

but you really shouldn't do that. You could also clone the repository and extract it. Then run

python setup.py build
python setup.py install

Connect to redis

The following code goes in your main.py or equivalent

redis_conn = tornadoredis.Client('hostname', 'port')
redis_conn.connect()

redis.connect is called only once. It is a blocking call, so it should be called before starting the main ioloop. The same connection object is shared between all the handlers.

You could add it to your application settings like

settings = {
    redis = redis_conn
}
app = tornado.web.Application([('/.*', Handler),],
                              **settings)

Use tornadoredis

The connection can be used in handlers as self.settings['redis'] or it can be added as a property of the BaseHandler class. Your request handlers subclass that class and access the property.

class BaseHandler(tornado.web.RequestHandler):

    @property
    def redis():
        return self.settings['redis']

To communicate with redis, the tornado.web.asynchronous and the tornado.gen.engine decorators are used

class SomeHandler(BaseHandler):

    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        foo = yield gen.Task(self.redis.get, 'foo')
        self.render('sometemplate.html', {'foo': foo}

Extra information

More examples and other features like connection pooling and pipelines can be found at the github repo.