Mutating Objects with shelve

Credit: Luther Blissett

Problem

You are using the standard module shelve, some of the values you have shelved are mutable objects, and you need to mutate these objects.

Solution

The shelve module, which offers a kind of persistent dictionary, occupies an important niche between the power of relational-database engines and the simplicity of marshal, pickle, dbm, and similar file formats. However, there’s a typical trap that you need to avoid when using shelve. Consider the following:

>>> import shelve
>>> # Build a simple sample shelf
>>> she=shelve.open('try.she', 'c')
>>> for c in 'spam': she[c]={c:23}
...
>>> for c in she.keys(  ): print c, she[c]
...
p {'p': 23}
s {'s': 23}
a {'a': 23}
m {'m': 23}
>>> she.close(  )

We’ve created the shelve object, added some data to it, and closed it. Now we can reopen it and work with it:

>>> she=shelve.open('try.she','c')
>>> she['p']
{'p': 23}
>>> she['p']['p'] = 42
>>> she['p']
{'p': 23}

What’s going on here? We just set the value to 42, but it didn’t take in the shelve object. The problem is that we were working with a temporary object that shelve gave us, but shelve doesn’t track changes to the temporary object. The solution is to bind a name to this temporary object, do our mutation, and then assign the mutated object back to the appropriate item of shelve:

>>> a = she['p']
>>> a['p'] = 42
>>> she['p'] = a
>>> she['p']
{'p': 42}
>>> she.close(  )

We can even verify the change:

>>> she=shelve.open('try.she','c')
>>> for c in she.keys(  ): print c,she[c]
...
p {'p': 42}
s {'s': 23}
a {'a': 23}
m {'m': 23}

Discussion

The standard Python module shelve can be quite convenient in many cases, but it hides a potentially nasty trap, which I could not find documented anywhere. Suppose you’re shelving mutable objects, such as dictionaries or lists. Naturally, you will want to mutate some of those objects—for example, by calling mutating methods (append on a list, update on a dictionary, and so on), or by assigning a new value to an item or attribute of the object. However, when you do this, the change doesn’t occur in the shelve object. This is because we are actually mutating a temporary object that the shelve object has given us as the result of its _ _getitem_ _ method, but the shelve object does not keep track of that temporary object, nor does it care about it once it returns it.

As shown in the recipe, the solution is to bind a name to the temporary object obtained by keying into the shelf, do whatever mutations are needed to the object via the name, then assign the newly mutated object back to the appropriate item of the shelve object. When you assign to a shelve item, the _ _setitem_ _ method is invoked, and it appropriately updates the shelve object itself, so that the change does occur.

See Also

Recipe 8.2 and Recipe 8.3 for alternative serialization approaches; documentation for the shelve standard library module in the Library Reference.

Get Python Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.