django request.POST contains <could not parse>

Question

I am having a django form to get the username,password. when the user posts data, i see that the post dictionary contains the following(traceback), Traceback (most recent call last):

File "/usr/lib/python2.4/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/usr/lib/python2.4/site-packages/django/views/decorators/csrf.py", line 39, in wrapped_view
resp = view_func(*args, **kwargs)

File "/usr/lib/python2.4/site-packages/django/views/decorators/csrf.py", line 52, in wrapped_view
return view_func(*args, **kwargs)

File "/public/gdp/trunk/src/ukl/lis/process/utils/error_handler.py", line 17, in __call__
return self.function(*args, **kwargs)

File "/usr/lib/python2.4/site-packages/django/views/decorators/cache.py", line 66, in _cache_controlled
response = viewfunc(request, *args, **kw)

File "/public/gdp/trunk/src/ukl/lis/process/authentication/views.py", line 530, in process_login
form = loginForm(request.POST)

File "/usr/lib/python2.4/site-packages/django/core/handlers/modpython.py", line 101, in _get_post
self._load_post_and_files()

File "/usr/lib/python2.4/site-packages/django/http/__init__.py", line 270, in _load_post_and_files
if self.META.get('CONTENT_TYPE', '').startswith('multipart'):

AttributeError: 'NoneType' object has no attribute 'startswith'

<ModPythonRequest
path:/login.html,
GET:<QueryDict: {}>,
POST:<could not parse>,
COOKIES:{'__utma': '115966011.1553834174.1346687405.1346687405.1346687045.1',
'__utmb': '115962011.4.10.1346687045',},
META:{'AUTH_TYPE': None,
'CONTENT_LENGTH': '85',
'CONTENT_TYPE': None,
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'text/html, application/xhtml+xml, */*',
'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
'HTTP_ACCEPT_LANGUAGE': 'en-GB',
'HTTP_CACHE_CONTROL': 'no-cache',
'HTTP_CONNECTION': 'Keep-Alive',
'HTTP_CONTENT_LENGTH': '85',
'HTTP_COOKIE': 'flavor=desktop; sessionid=4a2f2ab6f61315493f3038338524cfc7;    tmsid=e7c921af-9cae-4f58-8825-13f9bc2ba95f; uniqid=6f69c607-6aca-4e92-a112-b83691805155; __utma=115962011.1553833174.1346687005.1346687005.1346687005.1; __utmb=115962011.4.10.1346687005; __utmz=115962011.1346687005.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=115962011',
'HTTP_HOST': 'example.com',
'HTTP_REFERER': 'http://example.com/',
'HTTP_USER_AGENT': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64;  Trident/5.0)',
'PATH_INFO': u'/login.html',
'PATH_TRANSLATED': None,
'QUERY_STRING': None,
'REMOTE_ADDR': 'xx.xx.xx.xx',
'REMOTE_HOST': None,
'REMOTE_IDENT': None,
'REMOTE_USER': None,
'REQUEST_METHOD': 'POST',
'SCRIPT_NAME': '',
'SERVER_NAME': 'example.com',
'SERVER_PORT': 443,
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SOFTWARE': 'mod_python'}>

Why is the post dict has "<could not parse>" and why is CONTENT_TYPE None? This prevents me from crossing the login page.

This is my form,

   <form method="POST">{% csrf_token %}
       <tr>
     <td><label>Email Id</label></td>
     <td>{{form.username}}</td>
     <td><label>Password</label></td>
     <td>{{form.password}}</td>
   </tr>
    <input type="submit" name="btn_login" id="btn_login" class="btn_login" value="Login" /></td>

  </form>

Forms.py:

class loginForm(ModelForm):
    username = forms.CharField(max_length=50)
    password = forms.CharField(max_length=50, widget=forms.PasswordInput(attrs=  {"autocomplete":"off"}))

    class Meta:
        model = User

View:

def login(request):
 if request.method == "POST":
   log.inof(request)
 else:
  request.session.set_test_cookie()
  form = loginForm()
  return render_to_response('login.html', {'form':form},context_instance=RequestContext(request))

NOTE: I also found that all requests are coming from only mobile devices especially Blackberry. Seems one more user has the same problem

REF: https://stackoverflow.com/questions/12471661/mod-python-could-not-parse-the-django-post-request-for-blackberry-and-some-andro

Answer 1

Since the error doesn't occur on your development stack, just sometimes on production I doubt it is something in your Django code.

How are you contacting the machine? The requests is going to localhost.. Are you running a browser on your production machine or is there some proxy or load balancer in front of it?

It appears to me that something is mangling your requests before they hit mod_python, be it a proxy or misconfigured ssl.

The symptom is you receive broken requests. The problem could be anywhere from your python app down all OSI layers on your end, up all OSI layers on the clients end. Try to make full picture of all things in between and systematically rule them out.

Your TCP/IP stack seems to work fine, or you would see other requests and services suffer too. So we can rule out everything below the Session Layer. That leaves the Presentation Layer (SSL) and the Application Layer (HTTP).

1. SSL

   SERVER_PORT is 443, SSL is in play here. The request headers (HTTP_REFERER etc.) contain valid strings, so SSL seems to work fine. But I've seen some weirdly messed up TLS sessions before.

2. HTTP

   The players at this layer:

   • your Apache httpd
   • any reverse proxies on your end
   • any forward proxies between you and client. Are REMOTE_ADDR all in the same subnet, is there some mobile operator out there with broken proxies?
   • the client user agent. Are all blackberries broken? Try to get your hands on the browser you see failing most.

And at what layer are your loadbalancers operating?

Seems the main problem is a missing Content-Type header on the POST request, and hence a value on the Python representation of the POSTed content that can't be pretty printed using pprint, let alone used by your app.

Disregarding what I found in the Blackberry developer ref: have you tried setting enctype explicitly as Simon suggested? Try varying application/x-www-form-urlencoded and multipart/form-data.

Best of all try to reproduce it yourself. Try making a tcpdump that records such a request. You can analyse a recorded dump with wireshark.

Answer 2

Could you try to deploy with mod_wsgi instead of mod_python? I would start there, its probably not a django error or else everyone would be getting the error and not just the crackberry users. Mod_python is old and I've had many more issues with it that mod_wsgi.

Answer 3

django/core/handlers/wsgi.py seems to put the message there; you could hack the code and put the str(exception) there too.

My guess however is that you did not specify the charset correctly; django assuming UTF-8, and the form is submitting in some completely random charset that python cannot decode, perhaps? Also, you might want to try removing the CSRF middleware for a time and try if it works without it.

Answer 4

Maybe Blackberry clients gzip their POST bodies to save bandwidth? Django does not automatically decode gzipped messages (at least in Django 1.3.1). In your view, try decoding the raw POST data:

import zlib
post_data = zlib.decompress(request.raw_post_data, 16+zlib.MAX_WBITS)

Usually, the HTTP_CONTENT_ENCODING HTTP header should be set, but maybe the load balancer strips it away accidentally?