Processing chunked encoded HTTP POST requests in python (or generic CGI under apache)

Question

I have a j2me client that would post some chunked encoded data to a webserver. I'd like to process the data in python. The script is being run as a CGI one, but apparently apache will refuse a chunked encoded post request to a CGI script. As far as I could see mod_python, WSGI and FastCGI are no go too.

I'd like to know if there is a way to have a python script process this kind of input. I'm open to any suggestion (e.g. a confoguration setting in apache2 that would assemble the chunks, a standalone python server that would do the same, etc.) I did quite a bit of googling and didn't find anything usable, which is quite strange.

I know that resorting to java on the server side would be a solution, but I just can't imagine that this can't be solved with apache + python.

Answer 1

I had the exact same problem a year ago with a J2ME client talking to a Python/Ruby backend. The only solution I found which doesn't require application or infrastructure level changes was to use a relatively unknown feature of mod_proxy.

Mod_proxy has the ability to buffer incoming (chunked) requests, and then rewrite them as a single request with a Content-Length header before passing them on to a proxy backend. The neat trick is that you can create a tiny proxy configuration which passes the request back to the same Apache server. i.e. Take an incoming chunked request on port 80, "dechunk" it, and then pass it on to your non-HTTP 1.1 compliant server on port 81.

I used this configuration in production for a little over a year with no problems. It looks a little something like this:

ProxyRequests Off

<Proxy http://example.com:81>
  Order deny,allow
  Allow from all
</Proxy>

<VirtualHost *:80>
  SetEnv proxy-sendcl 1
  ProxyPass / http://example.com:81/
  ProxyPassReverse / http://example.com:81/
  ProxyPreserveHost On
  ProxyVia Full

  <Directory proxy:*>
    Order deny,allow
    Allow from all
  </Directory>

</VirtualHost>

Listen 81

<VirtualHost *:81>
  ServerName example.com
  # Your Python application configuration goes here
</VirtualHost>

I've also got a full writeup of the problem and my solution detailed on my blog.

Answer 2

I'd say use the twisted framework for building your http listener. Twisted supports chunked encoding.

http://python.net/crew/mwh/apidocs/twisted.web.http._ChunkedTransferEncoding.html

Hope this helps.

Answer 3

Apache 2.2 mod_cgi works fine for me, Apache transparently unchunks the request as it is passed to the CGI application.

WSGI currently disallows chunked requests, and mod_wsgi does indeed block them with a 411 response. It's on the drawing board for WSGI 2.0. But congratulations on finding something that does chunk requests, I've never seen one before!

Answer 4

You can't do what you want with mod_python. You can do it with mod_wsgi if you are using version 3.0. You do however have to step outside of the WSGI 1.0 specification as WSGI effectively prohibits chunked request content.

Search for WSGIChunkedRequest in http://code.google.com/p/modwsgi/wiki/ChangesInVersion0300 for what is required.

Answer 5

Maybe it is a configuration issue? Django can be fronted with Apache by mod_python, WSGI and FastCGI and it can accept file uploads.