Is Twisted's Deferred the same as a Promise in JavaScript?

Question

I started using Twisted in a project that require asynchronous programming and the docs are pretty good.

So my question is, is a Deferred in Twisted the same as a Promi

Answer 1

The answer to your question is both Yes and No depending on why you're asking.

Yes:

Both a Twisted Deferred and a Javascript Promise implement a mechanism for queuing synchronous blocks of code to be run in a given order while being decoupled from other synchronous blocks of code.

No:

So Javascript's Promise is actually more similar to Python's Future, and the airy-fairy way to explain this is to talk about the Promise and the Resolver being combined to make a Deferred, and to state that this affects what you can do with the callbacks.

This is all very well and good in that it's accurate, however it doesn't really make anything any clearer, and without typing thousands of words where I'm almost guaranteed to make a mistake, I'm probably better quoting someone who knows a little something about Python.

Guido van Rossum on Deferreds:

Here's my attempt to explain Deferred's big ideas (and there are a lot of them) to advanced Python users with no previous Twisted experience. I also assume you have thought about asynchronous calls before. Just to annoy Glyph, I am using a 5-star system to indicate the importance of ideas, where 1 star is "good idea but pretty obvious" and 5 stars is "brilliant".

I am showing a lot of code snippets, because some ideas are just best expressed that way -- but I intentionally leave out lots of details, and sometimes I show code that has bugs, if fixing them would reduce understanding the idea behind the code. (I will point out such bugs.) I am using Python 3.

Notes specifically for Glyph: (a) Consider this a draft for a blog post. I'd be more than happy to take corrections and suggestions for improvements. (b) This does not mean I am going to change Tulip to a more Deferred-like model; but that's for a different thread.

Idea 1: Return a special object instead of taking a callback argument

When designing APIs that produce results asynchronously, you find that you need a system for callbacks. Usually the first design that comes to mind is to pass in a callback function that will be called when the async operation is complete. I've even seen designs where if you don't pass in a callback the operation is synchronous -- that's bad enough I'd give it zero stars. But even the one-star version pollutes all APIs with extra arguments that have to be passed around tediously. Twisted's first big idea then is that it's better to return a special object to which the caller can add a callback after receiving it. I give this three stars because from it sprout so many of the other good ideas. It is of course similar to the idea underlying the Futures and Promises found in many languages and libraries, e.g. Python's concurrent.futures (PEP 3148, closely following Java Futures, both of which are meant for a threaded world) and now Tulip (PEP 3156, using a similar design adapted for thread-less async operation).

Idea 2: Pass results from callback to callback

I think it's best to show some code first:

class Deferred:
    def __init__(self):
        self.callbacks = []
    def addCallback(self, callback):
        self.callbacks.append(callback)  # Bug here
    def callback(self, result):
        for cb in self.callbacks:
            result = cb(result)

The most interesting bits are the last two lines: the result of each callback is passed to the next. This is different from how things work in concurrent.futures and Tulip, where the result (once set) is fixed as an attribute of the Future. Here the result can be modified by each callback.

This enables a new pattern when one function returning a Deferred calls another one and transforms its result, and this is what earns this idea three stars. For example, suppose we have an async function that reads a set of bookmarks, and we want to write an async function that calls this and then sorts the bookmarks. Instead of inventing a mechanism whereby one async function can wait for another (which we will do later anyway :-), the second async function can simply add a new callback to the Deferred returned by the first one:

def read_bookmarks_sorted():
    d = read_bookmarks()
    d.addCallback(sorted)
    return d

The Deferred returned by this function represents a sorted list of bookmarks. If its caller wants to print those bookmarks, it must add another callback:

d = read_bookmarks_sorted()
d.addCallback(print)

In a world where async results are represented by Futures, this same example would require two separate Futures: one returned by read_bookmarks() representing the unsorted list, and a separate Future returned by read_bookmarks_sorted() representing the sorted list.

There is one non-obvious bug in this version of the class: if addCallback() is called after the Deferred has already fired (i.e. its callback() method was called) then the callback added by addCallback() will never be called. It's easy enough to fix this, but tedious, and you can look it up in the Twisted source code. I'll carry this bug through successive examples -- just pretend that you live in a world where the result is never ready too soon. There are other problems with this design too, but I'd rather call the solutions improvements than bugfixes.

Aside: Twisted's poor choices of terminology

I don't know why, but, starting with the project's own name, Twisted often rubs me the wrong way with its choice of names for things. For example, I really like the guideline that class names should be nouns. But 'Deferred' is an adjective, and not just any adjective, it's a verb's past participle (and an overly long one at that :-). And why is it in a module named twisted.internet?

Then there is 'callback', which is used for two related but distinct purposes: it is the preferred term used for a function that will be called when a result is ready, but it is also the name of the method you call to "fire" the Deferred, i.e. set the (initial) result.

Don't get me started on the neologism/portmanteau that is 'errback', which leads us to...

Idea 3: Integrated error handling

This idea gets only two stars (which I'm sure will disappoint many Twisted fans) because it confused me a lot. I've also noted that the Twisted docs have some trouble explaining how it works -- In this case particularly I found that reading the code was more helpful than the docs.

The basic idea is simple enough: what if the promise of firing the Deferred with a result can't be fulfilled? When we write

d = pod_bay_doors.open()
d.addCallback(lambda _: pod.launch())

how is HAL 9000 supposed to say "I'm sorry, Dave. I'm afraid I can't do that" ?

And even if we don't care for that answer, what should we do if one of the callbacks raises an exception?

Twisted's solution is to bifurcate each callback into a callback and an 'errback'. But that's not all -- in order to deal with exceptions raised by callbacks, it also introduces a new class, 'Failure'. I'd actually like to introduce the latter first, without introducing errbacks:

class Failure:
    def __init__(self):
        self.exception = sys.exc_info()[1]

(By the way, great class name. And I mean this, I'm not being sarcastic.)

Now we can rewrite the callback() method as follows:

def callback(self, result):
    for cb in self.callbacks:
        try:
            result = cb(result)
        except:
            result = Failure()

This in itself I'd give two stars; the callback can use isinstance(result, Failure) to tell regular results apart from failures.

By the way, in Python 3 it might be possible to do away with the separate Failure class encapsulating exceptions, and just use the built-in BaseException class. From reading the comments in the code, Twisted's Failure class mostly exists so that it can hold all the information returned by sys.exc_info(), i.e. exception class/type, exception instance, and traceback but in Python 3, exception objects already hold a reference to the traceback.There is some debug stuff that Twisted's Failure class does which standard exceptions don't, but still, I think most reasons for introducing a separate class have been addressed.

But let's not forget about the errbacks. We change the list of callbacks to a list of pairs of callback functions, and we rewrite the callback() method again, as follows:

def callback(self, result):
    for (cb, eb) in self.callbacks:
        if isinstance(result, Failure):
            cb = eb  # Use errback
        try:
            result = cb(result)
        except:
            result = Failure()

For convenience we also add an errback() method:

def errback(self, fail=None):
    if fail is None:
        fail = Failure()
    self.callback(fail)

(The real errback() function has a few more special cases, it can be called with either an exception or a Failure as argument, and the Failure class takes an optional exception argument to prevent it from using sys.exc_info(). But none of that is essential and it makes the code snippets more complicated.)

In order to ensure that self.callbacks is a list of pairs we must also update addCallback() (it still doesn't work right when called after the Deferred has fired):

def addCallback(self, callback, errback=None):
    if errback is None:
        errback = lambda r: r
    self.callbacks.append((callback, errback))

If this is called with just a callback function, the errback will be a dummy that passes the result (i.e. a Failure instance) through unchanged. This preserves the error condition for a subsequent error handler. To make it easy to add an error handler without also handling a regular resullt, we add addErrback(), as follows:

def addErrback(self, errback):
    self.addCallback(lambda r: r, errback)

Here, the callback half of the pair will pass the (non-Failure) result through unchanged to the next callback.

If you want the full motivation, read Twisted's Introduction to Deferreds; I'll just end by noting that an errback and substitute a regular result for a Failure just by returning a non-Failure value (including None).

Before I move on to the next idea, let me point out that there are more niceties in the real Deferred class. For example, you can specify additional arguments to be passed to the callback and errback. But in a pinch you can do this with lambdas, so I'm leaving it out, because the extra code for doing the administration doesn't elucidate the basic ideas.

Idea 4: Chaining Deferreds

This is a five-star idea! Sometimes it really is necessary for a callback to wait for an additional async event before it can produce the desired result. For example, suppose we have two basic async operations, read_bookmarks() and sync_bookmarks(), and we want a combined operation. If this was synchronous code, we could write:

def sync_and_read_bookmarks():
    sync_bookmarks()
    return read_bookmarks()

But how do we write this if all operations return Deferreds? With the idea of chaining, we can do it as follows:

def sync_and_read_bookmarks():
    d = sync_bookmarks()
    d.addCallback(lambda unused_result: read_bookmarks())
    return d

The lambda is needed because all callbacks are called with a result value, but read_bookmarks() takes no arguments.