Unit testing a python app that uses the requests library

Question

I am writing an application that performs REST operations using Kenneth Reitz\'s requests library and I\'m struggling to find a nice way to unit test these applications, bec

Answer 1

You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.

As a very simple example, consider this class which uses the requests library:

class MyReq(object):
    def doSomething(self):
        r = requests.get('https://api.github.com', auth=('user', 'pass'))
        return r.headers['content-type']

Here's a unit test that intercepts the call to requests.get and returns a specified result for testing:

import unittest
import requests
import myreq

from mocker import Mocker, MockerTestCase

class MyReqTests(MockerTestCase):
    def testSomething(self):
        # Create a mock result for the requests.get call
        result = self.mocker.mock()
        result.headers
        self.mocker.result({'content-type': 'mytest/pass'})

        # Use mocker to intercept the call to requests.get
        myget = self.mocker.replace("requests.get")
        myget('https://api.github.com', auth=('user', 'pass'))
        self.mocker.result(result)

        self.mocker.replay()

        # Now execute my code
        r = myreq.MyReq()
        v = r.doSomething()

        # and verify the results
        self.assertEqual(v, 'mytest/pass')
        self.mocker.verify()

if __name__ == '__main__':
    unittest.main()

When I run this unit test I get the following result:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK

Answer 2

using mocker like in srgerg's answer:

def replacer(method, endpoint, json_string):
    from mocker import Mocker, ANY, CONTAINS
    mocker = Mocker()
    result = mocker.mock()
    result.json()
    mocker.count(1, None)
    mocker.result(json_string)
    replacement = mocker.replace("requests." + method)
    replacement(CONTAINS(endpoint), params=ANY)
    self.mocker.result(result)
    self.mocker.replay()

For the requests library, this would intercept the request by method and endpoint you're hitting and replace the .json() on the response with the json_string passed in.

Answer 3

If you use specifically requests try httmock. It's wonderfully simple and elegant:

from httmock import urlmatch, HTTMock
import requests

# define matcher:
@urlmatch(netloc=r'(.*\.)?google\.com$')
def google_mock(url, request):
    return 'Feeling lucky, punk?'

# open context to patch
with HTTMock(google_mock):
    # call requests
    r = requests.get('http://google.com/')
print r.content  # 'Feeling lucky, punk?'

If you want something more generic (e.g. to mock any library making http calls) go for httpretty.

Almost as elegant:

import requests
import httpretty

@httpretty.activate
def test_one():
    # define your patch:
    httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                        body="Find the best daily deals")
    # use!
    response = requests.get('http://yipit.com')
    assert response.text == "Find the best daily deals"

HTTPretty is far more feature-rich - it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).

Answer 4

It is in fact a little strange that the library has a blank page about end-user unit testing, while targeting user-friendliness and ease of use. There's however an easy-to-use library by Dropbox, unsurprisingly called responses. Here is its intro post. It says they've failed to employ httpretty, while stating no reason of the fail, and written a library with similar API.

import unittest

import requests
import responses


class TestCase(unittest.TestCase):

  @responses.activate  
  def testExample(self):
    responses.add(**{
      'method'         : responses.GET,
      'url'            : 'http://example.com/api/123',
      'body'           : '{"error": "reason"}',
      'status'         : 404,
      'content_type'   : 'application/json',
      'adding_headers' : {'X-Foo': 'Bar'}
    })

    response = requests.get('http://example.com/api/123')

    self.assertEqual({'error': 'reason'}, response.json())
    self.assertEqual(404, response.status_code)

Answer 5

Missing from these answers is requests-mock.

From their page:

>>> import requests
>>> import requests_mock

As a context manager:

>>> with requests_mock.mock() as m:

...     m.get('http://test.com', text='data')
...     requests.get('http://test.com').text
...
'data'

Or as a decorator:

>>> @requests_mock.mock()
... def test_func(m):
...     m.get('http://test.com', text='data')
...     return requests.get('http://test.com').text
...
>>> test_func()
'data'