You are here: Home Articles Mock testing with mocker and plone.mocktestcase
Navigation
OpenID Log in

 

Mock testing with mocker and plone.mocktestcase

by Martin Aspeli last modified Jun 27, 2008 01:11 PM

You must be having a laugh

It is a well known fact that most tests people write for Plone projects are integration tests rather than unit tests. Using PloneTestCase, we bootstrap pretty much all of Zope and Plone to execute tests, and zap the transaction at the end of each test run to avoid persistent changes bleeding between tests.

Integration tests are important, and definitely have their place, but the most basic type of test should be a true unit test. A unit test should have no (or minimal) dependencies on anything outside the function or class under test. It should test inputs, outputs and edge cases. Whenever the code under test calls an external API, the unit test should assume that this code works as advertised (the obvious corollary being that the things it relies on have sensible and stable APIs), rather than attempt to test those as well. And most importantly, a unit test should be easy to understand, quick to write and fast to run.

It is easy to write true unit tests for simple functions that have well-defined inputs and outputs and do not make a lot of assumptions about their environment. However, most real-world applications are not like that. Consider the following function:

from plone.supermodel import load_string, load_file
from plone.supermodel.model import Model, FILENAME_KEY


...

    def lookup_model(self):
        
        if self.model_source:
            return load_string(self.model_source, policy=u"dexterity")
        
        elif self.model_file:
            model_file = self._abs_model_file()
            return load_file(model_file, reload=True, policy=u"dexterity")
        
        elif self.schema:
            
            # Attempt to load model file if it was tagged onto the schema
            # interface or one of its bases
            
            schema = self.lookup_schema()
            
            for iface in [schema] + list(schema.getBases()):
                model_file = iface.queryTaggedValue(FILENAME_KEY)
                if model_file is not None:
                    return load_file(model_file, reload=False)
            
            # Otherwise, return a model with just this interface
            
            return Model({u"": schema})
        
        raise ValueError("Neither model source, nor model file, nor schema is specified in FTI %s" % self.getId())

This looks innocent enough, and is in fact rather well abstracted. However, think about the inputs of this function. It takes no parameters, but its operation is affected by a number of variables - self.model_source, self.model_file and self.schema. It uses a few different internal helper functions - self.lookup_schema() and self._abs_model_file() - and a few external helper functions - load_file() and load_string(). And it returns a Model object or raises a ValueError.

To test unit this method, we want to be able to try:

  • Different combinations of "inputs", i.e. different settings for the variables self.model_source, self.model_file and self.schema
  • Different return values from the API methods load_file() and load_string()
  • Possibly, different return values from the internal helper functions, self.lookup_schema() and self._abs_model_file()

In each case, we want to observe the return value and/or look for any exceptions thrown.

In other cases, we may also want to make assertions about:

  • API calls that represent "outputs", i.e. actions that the code under test is expected to perform in certain situations.
  • The parameters passed to such API calls.
  • The behaviour of the code under test in cases where API methods throw exceptions or return unexpected values.

In the example above, API calls were simply global functions. When writing code for Zope or Plone, we often find that APIs are abstracted away behind interfaces and accessed as utilities (via a getUtility() call) or adapters (by "calling" an interface or calling a function like getMultiAdapter()).

The need for mock objects

In order to properly unit test the function above without depending on the specific behaviour of APIs that it uses, we need to provide mock-ups of these. It is often possible to simply create your own mock objects in Python code and pass these in or otherwise trick the code under test into using them. However, this gets cumbersome and is sometimes quite difficult. Furthermore, you often want to make an assertion of the type "expect this API to be called with these parameters". This can be cumbersome with bespoke mocks.

For this reason, a number of "mock object" libraries have sprung up. There are several implementations, but they all work in pretty much the same way.

First, you set up a number of mock objects, which are either closely or loosely tied to the "real" object or function that they emulate. You then set a number of expectations, of the form "ensure that this method is called with these particular parameters". You can also mock up return values (which themselves can be mocks), simulate the throwing of an exception, and declare how many times during a test run you expect a given mock to be invoked.

Once you have completed the "recording" of actions, you put the mock library into "replay" mode, and execute the method under test. The mock library will cause a test failure if a mock is invoked in an unexpected way, and will simulate return values and other behaviour as specified during mock recording.

Finally, the mock is verified (to ensure that everything you expected to be called, was in fact called) and restored (to undo any runtime patching that the mock libarary may have done).

Introducing mocker and plone.mocktestcase

There are a number of mock object libraries for Python, but Mocker seems to be one of the better ones. For one thing, it has pretty good documentation. For another, it supports some advanced features for mocking global functions, and for creating proxies that pass through to a "real" object in most cases, but still allow mock assertions to be made.

To make it easier to use Mocker with Plone, I have created a small integration library called plone.mocktestcase (which will be released to PyPI shortly so that you can depend on it). This can be used as a unittest.TestCase-style base class. In fact, Mocker already has such a base class, which takes care of verifying and restoring the mock during test tear-down. In addition, plone.mocktestcase's base class:

  • Provides helper functions to register mock objects as utilities or adapters.
  • Provides a method to make simple "dummies". A dummy is an object that has some pre-defined attributes or methods, but which does not carry mock assertions.
  • Provides a couple of "matchers" that can be used to make assertions about mock method parameters, including ones to test the type (class) and a provided interface of an parameter.
  • Tears down the Zope 3 Component Architecture after each test.

Most of the magic, however, comes from mocker itself. It allows three types of mock objects:

  • A "mock", created with self.mocker.mock(), is a simple mock object. You can pass a class to the mock() method to ensure that only methods and attributes of that class are called (otherwise, it'd be easy to have a mock get out of sync with the class it was trying to mock up).
  • A "proxy", created with self.mocker.proxy(obj). The proxy will pass through to the underlying object, but you can make mock assertions and provide mock return values.
  • A "replacement", created with self.mocker.replace("some.dotted.name"). This will actually replace the real implementation of the given object with a mock version. This is very powerful, because it allows you to mock global functions and classes much more easily. When the mock is restored, the original object is re-instated.

Here is an example of a test case using mocker. To learn more about the API, see the Mocker documentation.

import unittest
from plone.mocktestcase import MockTestCase

class TestFTI(MockTestCase):

    ...

    def test_lookup_model_from_string(self):
        fti = DexterityFTI(u"testtype")
        fti.schema = None
        fti.model_source = "<model />"
        fti.model_file = None
        
        model_dummy = Model()
        
        load_string_mock = self.mocker.replace("plone.supermodel.load_string")
        self.expect(load_string_mock(fti.model_source, policy=u"dexterity")).result(model_dummy)
        
        self.replay()
        
        model = fti.lookup_model()
        self.assertIs(model_dummy, model)

    ...

def test_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(TestFTI))
    return suite

 

The example above makes use of the replace() method to replace the global load_string() method with a mock. It expects it to be called with a particular string (in the fti.model_source variable) as the first parameter and the string u"dexterity" as the policy keyword argument. It will then return the model_dummy object, which we use in an assertion at the end of the test.

We must remember to put the test case into replay mode, using self.replay(), before calling the method under test, fti.lookup_model(). The test case takes care of everything else.

Here is another example of a mock based test:

    def test_concrete_default_schema(self):
        
        # Mock schema model
        class IDummy(Interface):
            dummy = zope.schema.TextLine(title=u"Dummy")
        mock_model = Model({u"": IDummy})
        
        # Mock FTI
        fti_mock = self.mocker.mock(DexterityFTI)
        self.expect(fti_mock.lookup_model()).result(mock_model)
        self.mock_utility(fti_mock, IDexterityFTI, u'testtype')
        
        self.mocker.replay()
        
        factory = schema.SchemaModuleFactory()
        
        schema_name = utils.portal_type_to_schema_name('testtype', prefix='site')
        klass = factory(schema_name, schema.generated)
        
        self.failUnless(isinstance(klass, InterfaceClass))
        self.failUnless(klass.isOrExtends(IDexteritySchema))
        self.failUnless(IContentType.providedBy(klass))
        self.assertEquals(schema_name, klass.__name__)
        self.assertEquals('plone.dexterity.schema.generated', klass.__module__)
        self.assertEquals(('dummy',), tuple(zope.schema.getFieldNames(klass)))

In this example, we create a mock that is expected to conform to the DexterityFTI class. We expect the code under test to call lookup_model() on this, and will return a dummy Model object when it does so.

We then register this mock object as a utility providing IDexterityFTI under the name u"testtype". This registration will only last for the duration of the test method, since the Component Architecture is torn down after each test run. Thus, we do not need to worry about cleanup.

Here is an example of using a proxy:

    def test_form_fields(self):
        
        # The 'fields' property on the form should look up the schema from
        # the FTI utility with the name given as the portal_type.
        
        # Context and request
        context_mock = self.mocker.mock()
        request_mock = self.mocker.mock()
        
        # Schema
        
        schema_mock = self.mocker.mock()
        
        # Form fields generator
        
        fields_dummy = object()
        fields_mock = self.mocker.replace('z3c.form.field.Fields')
        self.expect(fields_mock(schema_mock)).result(fields_dummy)
        
        # FTI
        
        fti_mock = self.mocker.proxy(DexterityFTI(u"testtype"))
        self.expect(fti_mock.lookup_schema()).result(schema_mock)
        self.mock_utility(fti_mock, IDexterityFTI, name=u"testtype")
        
        self.replay()
        
        form = DefaultAddForm(context_mock, request_mock, u"testtype")
        self.assertIs(fields_dummy, form.fields)
.portal_type)

We first replace the global z3c.form.field.Fields constructor with a mock version that returns dummy fields. We then create a mocker proxy for a new instance of a DexterityFTI object, and set the expectation that lookup_schema() should be called on it once, returning the schema_mock, a simple mock. Any other attribute of the proxy mock will pass through to the underlying object without assertions. Again, we register this mock as a utility.

Other types of assertions and operations include:

  • Ensure that a function is called at least N times and/or at most M times, using mock.count(N,M).
  • Ensure that a function is not called, using mock.count(0).
  • Ensure that a function is called with a parameter providing interface IFoo, with self.expect(mock.func(self.match_provides(IFoo))).

Finally, sometimes we just want to have an object with a few properties. For example, let's say we want to mock up a context that has a portal_type property, and we do not care how many times it is called. With a mock, this would be done as:

context_mock = self.mocker.mock()
self.expect(context_mock.portal_type).result(u"type").count(0,None)

However, this is a bit cumbersome. Instead, you can create a dummy object. The dummy is not nearly as clever as a mock, but it is a little easier to create. For example:

context_dummy = self.create_dummy(portal_type=u"type")

 You can see many more examples of mock-based tests in the plone.dexterity package.

Tips and Caveats

Mock testing can be very powerful, allowing very fine grained control over tests. Mock tests are also very quick to execute, and relatively easy to write once you get the hang of it.

There are two big dangers with mock testing, however. First of all, you may end up being overly reliant on the internals of the code you are testing. Good mock tests will not make too many assumptions about implementation detail, instead being written in terms of APIs that have semantic meaning and documented behaviour. Well-written code is usually easier to unit test, especially with mocks. When writing new code, is often useful to think "how would I unit test this?". If the answer is "I couldn't", the code is probably poorly designed.

Secondly, it is sometimes easy to create mocks that get out of sync with the real world. Writing an accurate mock usually requires a pretty good understanding of the thing you are mocking. That's not necessarily a bad thing, but if you are the type of test author who fiddles the test until it works, it's quite easy to make a mock behave however you want, regardless of how the real object would behave.

With this in mind, here are a few suggestions:

  • Don't stop writing integration tests! Most projects will require both unit and integration tests.
    • Use (slow) integration tests to test the major flows through your code.
    • Use (fast) mock-based unit tests (or simple non-mock unit test if you can) to test edge cases and detailed behaviour.
  • When creating a mock using self.mocker.mock(), pass in the class of the object you are mocking. This lets the library stop you from mocking up things that don't exist on the class you are mocking up.
  • Do not mock up simple data structures or "value objects". If an object can simply be instantiated and passed to the code under test, then do that rather than creating a mock for the sake of it.
  • Make sure your code is properly abstracted and uses well-documented, stable APIs. This makes mocking much easier.
  • Don't mock the getUtility() call or an adapter's provided interface. Instead, create a mock object for the component and register it using self.mock_utility() or self.mock_adapter().
Document Actions
Plone Book
Professional Plone 4 Development

I am the author of a book called Professional Plone Development. You can read more about it here.

About this site

This Plone site is kindly hosted by: 

Six Feet Up