I got the following to work already:
You don't need the richfaces a4j:commandLink tag for the download, the standard jsf tag h:commandLink would be enough.
Then make sure you have the following headers set on your response (you can check with firebug in firefox):
Content-Disposition attachment; filename="your_file_name.xxx"
Content-Type application/xxx
Content-Length 1234
Content-Length: number of bytes.