Python Cookbook

Synchronizing All Methods in an Object

Credit: André Bjärby

Problem

You want to share an object among multiple threads, but to avoid conflicts you need to ensure that only one thread at a time is inside the object, possibly excepting some methods for which you want to hand-tune locking behavior.

Solution

Java offers such synchronization as a built-in feature, while in Python you have to program it explicitly using reentrant locks, but this is not all that hard:

import types

def _get_method_names(obj):
    """ Get all methods of a class or instance, inherited or otherwise. """
    if type(obj) == types.InstanceType:
        return _get_method_names(obj._ _class_ _)
    elif type(obj) == types.ClassType:
        result = []
        for name, func in obj._ _dict_ _.items(  ):
            if type(func) == types.FunctionType:
                result.append((name, func))
        for base in obj._ _bases_ _:
            result.extend(_get_method_names(base))
        return result


class _SynchronizedMethod:
    """ Wrap lock and release operations around a method call. """

    def _ _init_ _(self, method, obj, lock):
        self._ _method = method
        self._ _obj = obj
        self._ _lock = lock

    def _ _call_ _(self, *args, **kwargs):
        self._ _lock.acquire(  )
        try:
            return self._ _method(self._ _obj, *args, **kwargs)
        finally:
            self._ _lock.release(  )


class SynchronizedObject:
    """ Wrap all methods of an object into _SynchronizedMethod instances. """

    def _ _init_ _(self, obj, ignore=[], lock=None):
        import threading

        # You must access _ _dict_ _ directly to avoid tickling _ _setattr_ _
        self._ _dict_ _['_SynchronizedObject_ _methods'] = {}
        self._ _dict_ _['_SynchronizedObject_ _obj'] = obj
        if not lock: lock = threading.RLock(  )
        for name, method in _get_method_names(obj):
            if not name in ignore and not self._ _methods.has_key(name):
                self._ _methods[name] = _SynchronizedMethod(method, obj, lock)

    def _ _getattr_ _(self, name):
        try:
            return self._ _methods[name]
        except KeyError:
            return getattr(self._ _obj, name)

    def _ _setattr_ _(self, name, value):
        setattr(self._ _obj, name, value)

Discussion

As usual, we complete this module with a small self test, executed only when the module is run as main script. This also serves to show how the module’s functionality can be used:

if _ _name_ _ == '_ _main_ _':
    import threading
    import time

    class Dummy:

        def foo (self):
            print 'hello from foo'
            time.sleep(1)

        def bar (self):
            print 'hello from bar'

        def baaz (self):
            print 'hello from baaz'

    tw = SynchronizedObject(Dummy(  ), ignore=['baaz'])
    threading.Thread(target=tw.foo).start(  )
    time.sleep(.1)
    threading.Thread(target=tw.bar).start(  )
    time.sleep(.1)
    threading.Thread(target=tw.baaz).start(  )

Thanks to the synchronization, the call to bar runs only when the call to foo has completed. However, because of the ignore= keyword argument, the call to baaz bypasses synchronization and thus completes earlier. So the output is:

hello from foo
hello from baaz
hello from bar

When you find yourself using the same single-lock locking code in almost every method of an object, use this recipe to refactor the locking away from the object’s application-specific logic. The key code idiom is:

self.lock.acquire(  )
try:
   # The "real" application code for the method
finally:
    self.lock.release(  )

To some extent, this recipe can also be handy when you want to postpone worrying about a class’s locking behavior. Note, however, that if you intend to use this code for production purposes, you should understand all of it. In particular, this recipe is not wrapping direct accesses, be they get or set, to the object’s attributes. If you also want them to respect the object’s lock, you’ll need the object you’re wrapping to define, in turn, its own _ _getattr_ _ and _ _setattr_ _ special methods.

This recipe is carefully coded to work with every version of Python, including old ones such as 1.5.2, as long as you’re wrapping classic classes (i.e., classes that don’t subclass built-in types). Issues, as usual, are subtly different for Python 2.2 new-style classes (which subclass built-in types or the new built-in type object that is now the root class). Metaprogramming (e.g., the tasks performed in this recipe) sometimes requires a subtly different approach when you’re dealing with the new-style classes of Python 2.2 and later.

See Also

Documentation of the standard library modules threading and types in the Library Reference.