Mutation Tracking

Mutation Tracking

Provide support for tracking of in-place changes to scalar values,which are propagated into ORM change events on owning parent objects.

Establishing Mutability on Scalar Column Values

A typical example of a “mutable” structure is a Python dictionary.Following the example introduced in Column and Data Types, webegin with a custom type that marshals Python dictionaries intoJSON strings before being persisted:

from sqlalchemy.types import TypeDecorator, VARCHAR
import json
 
class JSONEncodedDict(TypeDecorator):
    "Represents an immutable structure as a json-encoded string."
 
    impl = VARCHAR
 
    def process_bind_param(self, value, dialect):
        if value is not None:
            value = json.dumps(value)
        return value
 
    def process_result_value(self, value, dialect):
        if value is not None:
            value = json.loads(value)
        return value

The usage of json is only for the purposes of example. Thesqlalchemy.ext.mutable extension can be usedwith any type whose target Python type may be mutable, includingPickleType, postgresql.ARRAY, etc.

When using the sqlalchemy.ext.mutable extension, the value itselftracks all parents which reference it. Below, we illustrate a simpleversion of the MutableDict dictionary object, which appliesthe Mutable mixin to a plain Python dictionary:

from sqlalchemy.ext.mutable import Mutable
 
class MutableDict(Mutable, dict):
    @classmethod
    def coerce(cls, key, value):
        "Convert plain dictionaries to MutableDict."
 
        if not isinstance(value, MutableDict):
            if isinstance(value, dict):
                return MutableDict(value)
 
            # this call will raise ValueError
            return Mutable.coerce(key, value)
        else:
            return value
 
    def __setitem__(self, key, value):
        "Detect dictionary set events and emit change events."
 
        dict.__setitem__(self, key, value)
        self.changed()
 
    def __delitem__(self, key):
        "Detect dictionary del events and emit change events."
 
        dict.__delitem__(self, key)
        self.changed()

The above dictionary class takes the approach of subclassing the Pythonbuilt-in dict to produce a dictsubclass which routes all mutation events through setitem. There arevariants on this approach, such as subclassing UserDict.UserDict orcollections.MutableMapping; the part that’s important to this example isthat the Mutable.changed() method is called whenever an in-placechange to the datastructure takes place.

We also redefine the Mutable.coerce() method which will be used toconvert any values that are not instances of MutableDict, suchas the plain dictionaries returned by the json module, into theappropriate type. Defining this method is optional; we could just as wellcreated our JSONEncodedDict such that it always returns an instanceof MutableDict, and additionally ensured that all calling codeuses MutableDict explicitly. When Mutable.coerce() is notoverridden, any values applied to a parent object which are not instancesof the mutable type will raise a ValueError.

Our new MutableDict type offers a class methodas_mutable() which we can use within column metadatato associate with types. This method grabs the given type object orclass and associates a listener that will detect all future mappingsof this type, applying event listening instrumentation to the mappedattribute. Such as, with classical table metadata:

from sqlalchemy import Table, Column, Integer
 
my_data = Table('my_data', metadata,
    Column('id', Integer, primary_key=True),
    Column('data', MutableDict.as_mutable(JSONEncodedDict))
)

Above, as_mutable() returns an instance of JSONEncodedDict(if the type object was not an instance already), which will intercept anyattributes which are mapped against this type. Below we establish a simplemapping against the my_data table:

from sqlalchemy import mapper
 
class MyDataClass(object):
    pass
 
# associates mutation listeners with MyDataClass.data
mapper(MyDataClass, my_data)

The MyDataClass.data member will now be notified of in place changesto its value.

There’s no difference in usage when using declarative:

from sqlalchemy.ext.declarative import declarative_base
 
Base = declarative_base()
 
class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(MutableDict.as_mutable(JSONEncodedDict))

Any in-place changes to the MyDataClass.data memberwill flag the attribute as “dirty” on the parent object:

>>> from sqlalchemy.orm import Session
 
>>> sess = Session()
>>> m1 = MyDataClass(data={'value1':'foo'})
>>> sess.add(m1)
>>> sess.commit()
 
>>> m1.data['value1'] = 'bar'
>>> assert m1 in sess.dirty
True

The MutableDict can be associated with all future instancesof JSONEncodedDict in one step, usingassociate_with(). This is similar toas_mutable() except it will intercept all occurrencesof MutableDict in all mappings unconditionally, withoutthe need to declare it individually:

MutableDict.associate_with(JSONEncodedDict)
 
class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(JSONEncodedDict)

Supporting Pickling

The key to the sqlalchemy.ext.mutable extension relies upon theplacement of a weakref.WeakKeyDictionary upon the value object, whichstores a mapping of parent mapped objects keyed to the attribute name underwhich they are associated with this value. WeakKeyDictionary objects arenot picklable, due to the fact that they contain weakrefs and functioncallbacks. In our case, this is a good thing, since if this dictionary werepicklable, it could lead to an excessively large pickle size for our valueobjects that are pickled by themselves outside of the context of the parent.The developer responsibility here is only to provide a getstate methodthat excludes the _parents() collection from the picklestream:

class MyMutableType(Mutable):
    def __getstate__(self):
        d = self.__dict__.copy()
        d.pop('_parents', None)
        return d

With our dictionary example, we need to return the contents of the dict itself(and also restore them on setstate):

class MutableDict(Mutable, dict):
    # ....
 
    def __getstate__(self):
        return dict(self)
 
    def __setstate__(self, state):
        self.update(state)

In the case that our mutable value object is pickled as it is attached to oneor more parent objects that are also part of the pickle, the Mutablemixin will re-establish the Mutable._parents collection on each valueobject as the owning parents themselves are unpickled.

Receiving Events

The AttributeEvents.modified() event handler may be used to receivean event when a mutable scalar emits a change event. This event handleris called when the attributes.flag_modified() function is calledfrom within the mutable extension:

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
 
Base = declarative_base()
 
class MyDataClass(Base):
    __tablename__ = 'my_data'
    id = Column(Integer, primary_key=True)
    data = Column(MutableDict.as_mutable(JSONEncodedDict))
 
@event.listens_for(MyDataClass.data, "modified")
def modified_json(instance):
    print("json value modified:", instance.data)

Establishing Mutability on Composites

Composites are a special ORM feature which allow a single scalar attribute tobe assigned an object value which represents information “composed” from oneor more columns from the underlying mapped table. The usual example is that ofa geometric “point”, and is introduced in Composite Column Types.

As is the case with Mutable, the user-defined composite classsubclasses MutableComposite as a mixin, and detects and deliverschange events to its parents via the MutableComposite.changed() method.In the case of a composite class, the detection is usually via the usage ofPython descriptors (i.e. @property), or alternatively via the specialPython method setattr(). Below we expand upon the Point classintroduced in Composite Column Types to subclass MutableCompositeand to also route attribute set events via setattr to theMutableComposite.changed() method:

from sqlalchemy.ext.mutable import MutableComposite
 
class Point(MutableComposite):
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
    def __setattr__(self, key, value):
        "Intercept set events"
 
        # set the attribute
        object.__setattr__(self, key, value)
 
        # alert all parents to the change
        self.changed()
 
    def __composite_values__(self):
        return self.x, self.y
 
    def __eq__(self, other):
        return isinstance(other, Point) and \
            other.x == self.x and \
            other.y == self.y
 
    def __ne__(self, other):
        return not self.__eq__(other)

The MutableComposite class uses a Python metaclass to automaticallyestablish listeners for any usage of orm.composite() that specifies ourPoint type. Below, when Point is mapped to the Vertex class,listeners are established which will route change events from Pointobjects to each of the Vertex.start and Vertex.end attributes:

from sqlalchemy.orm import composite, mapper
from sqlalchemy import Table, Column
 
vertices = Table('vertices', metadata,
    Column('id', Integer, primary_key=True),
    Column('x1', Integer),
    Column('y1', Integer),
    Column('x2', Integer),
    Column('y2', Integer),
    )
 
class Vertex(object):
    pass
 
mapper(Vertex, vertices, properties={
    'start': composite(Point, vertices.c.x1, vertices.c.y1),
    'end': composite(Point, vertices.c.x2, vertices.c.y2)
})

Any in-place changes to the Vertex.start or Vertex.end memberswill flag the attribute as “dirty” on the parent object:

>>> from sqlalchemy.orm import Session
 
>>> sess = Session()
>>> v1 = Vertex(start=Point(3, 4), end=Point(12, 15))
>>> sess.add(v1)
>>> sess.commit()
 
>>> v1.end.x = 8
>>> assert v1 in sess.dirty
True

Coercing Mutable Composites

The MutableBase.coerce() method is also supported on composite types.In the case of MutableComposite, the MutableBase.coerce()method is only called for attribute set operations, not load operations.Overriding the MutableBase.coerce() method is essentially equivalentto using a validates() validation routine for all attributes whichmake use of the custom composite type:

class Point(MutableComposite):
    # other Point methods
    # ...
 
    def coerce(cls, key, value):
        if isinstance(value, tuple):
            value = Point(*value)
        elif not isinstance(value, Point):
            raise ValueError("tuple or Point expected")
        return value

Supporting Pickling

As is the case with Mutable, the MutableComposite helperclass uses a weakref.WeakKeyDictionary available via theMutableBase._parents() attribute which isn’t picklable. If we need topickle instances of Point or its owning class Vertex, we at least needto define a getstate that doesn’t include the parents dictionary.Below we define both a getstate and a _setstate that package upthe minimal form of our Point class:

class Point(MutableComposite):
    # ...
 
    def __getstate__(self):
        return self.x, self.y
 
    def __setstate__(self, state):
        self.x, self.y = state

As with Mutable, the MutableComposite augments thepickling process of the parent’s object-relational state so that theMutableBase._parents() collection is restored to all Point objects.

API Reference

class sqlalchemy.ext.mutable.MutableBase
Common base class to Mutableand MutableComposite.
- _parents
- Dictionary of parent object->attribute name on the parent.

This attribute is a so-called “memoized” property. It initializesitself with a new weakref.WeakKeyDictionary the first timeit is accessed, returning the same object upon subsequent access.

classmethod coerce(key, value)
Given a value, coerce it into the target type.

Can be overridden by custom subclasses to coerce incomingdata into a particular type.

By default, raises ValueError.

This method is called in different scenarios depending on ifthe parent class is of type Mutable or of typeMutableComposite. In the case of the former, it is calledfor both attribute-set operations as well as during ORM loadingoperations. For the latter, it is only called during attribute-setoperations; the mechanics of the composite() constructhandle coercion during load operations.

- Parameters
- 
  -

key – string name of the ORM-mapped attribute being set.

value – the incoming value.

- Returns
-

the method should return the coerced value, or raiseValueError if the coercion cannot be completed.

class sqlalchemy.ext.mutable.Mutable
Bases: sqlalchemy.ext.mutable.MutableBase

Mixin that defines transparent propagation of changeevents to a parent object.

See the example in Establishing Mutability on Scalar Column Values for usage information.

eq()

inherited from the eq() method of object

Return self==value.

init()

inherited from the init() method of object

Initialize self. See help(type(self)) for accurate signature.

le()

inherited from the le() method of object

Return self<=value.

lt()

inherited from the lt() method of object

Return selfne()="" -="" inherited="" from="" the="" ne()="" method="" of="" object="" return="" self!="value." -="" classmethod="" get_listen_keys(_attribute)="" -="" inherited="" from="" the="" get_listen_keys()="" _method="" ofMutableBase="" given="" a="" descriptor="" attribute,="" return="" a="" set()="" of="" the="" attributekeys="" which="" indicate="" a="" change="" in="" the="" state="" of="" this="" attribute.="" this="" is="" normally="" just="" set([attribute.key]),="" but="" can="" be="" overriddento="" provide="" for="" additional="" keys.="" e.g.="" a="" MutableCompositeaugments="" this="" set="" with="" the="" attribute="" keys="" associated="" with="" the="" columnsthat="" comprise="" the="" composite="" value.="" this="" collection="" is="" consulted="" in="" the="" case="" of="" intercepting="" theInstanceEvents.refresh()="" andInstanceEvents.refresh_flush()="" events,="" which="" pass="" along="" a="" listof="" attribute="" names="" that="" have="" been="" refreshed;="" the="" list="" is="" comparedagainst="" this="" set="" to="" determine="" if="" action="" needs="" to="" be="" taken.="" new="" in="" version="" 1.0.5.="" -="" classmethod="" listen_on_attribute(_attribute,="" coerce,="" parent_cls)="" -="" inherited="" from="" the="" listen_on_attribute()="" _method="" ofMutableBase="" establish="" this="" type="" as="" a="" mutation="" listener="" for="" the="" givenmapped="" descriptor.="" -="" parents="" -="" _inherited="" from="" the_parentsattribute="" ofMutableBase="" dictionary="" of="" parent="" object-="">attribute name on the parent.

This attribute is a so-called “memoized” property. It initializesitself with a new weakref.WeakKeyDictionary the first timeit is accessed, returning the same object upon subsequent access.

classmethod asmutable(_sqltype)
Associate a SQL type with this mutable Python type.

This establishes listeners that will detect ORM mappings againstthe given type, adding mutation event trackers to those mappings.

The type is returned, unconditionally as an instance, so thatas_mutable() can be used inline:

Table('mytable', metadata,
    Column('id', Integer, primary_key=True),
    Column('data', MyMutableType.as_mutable(PickleType))
)

Note that the returned type is always an instance, even if a classis given, and that only columns which are declared specifically withthat type instance receive additional instrumentation.

To associate a particular mutable type with all occurrences of aparticular type, use the Mutable.associate_with() classmethodof the particular Mutable subclass to establish a globalassociation.

Warning

The listeners established by this method are global_to all mappers, and are _not garbage collected. Only useas_mutable() for types that are permanent to an application,not with ad-hoc types else this will cause unbounded growthin memory usage.

classmethod associatewith(_sqltype)
Associate this wrapper with all future mapped columnsof the given type.

This is a convenience method that callsassociate_with_attribute automatically.

Warning

The listeners established by this method are global_to all mappers, and are _not garbage collected. Only useassociate_with() for types that are permanent to anapplication, not with ad-hoc types else this will cause unboundedgrowth in memory usage.

classmethod associatewith_attribute(_attribute)
Establish this type as a mutation listener for the givenmapped descriptor.
changed()
Subclasses should call this method whenever change events occur.
classmethod coerce(key, value)

inherited from thecoerce()method ofMutableBase

Given a value, coerce it into the target type.

Can be overridden by custom subclasses to coerce incomingdata into a particular type.

By default, raises ValueError.

- Parameters
- 
  -

key – string name of the ORM-mapped attribute being set.

value – the incoming value.

- Returns
-

the method should return the coerced value, or raiseValueError if the coercion cannot be completed.

class sqlalchemy.ext.mutable.MutableComposite
Bases: sqlalchemy.ext.mutable.MutableBase

Mixin that defines transparent propagation of changeevents on a SQLAlchemy “composite” object to itsowning parent or parents.

See the example in Establishing Mutability on Composites for usage information.

changed()
Subclasses should call this method whenever change events occur.

class sqlalchemy.ext.mutable.MutableDict
Bases: sqlalchemy.ext.mutable.Mutable, builtins.dict

A dictionary type that implements Mutable.

The MutableDict object implements a dictionary that willemit change events to the underlying mapping when the contents ofthe dictionary are altered, including when values are added or removed.

Note that MutableDict does not apply mutable tracking to thevalues themselves inside the dictionary. Therefore it is not a sufficientsolution for the use case of tracking deep changes to a _recursive_dictionary structure, such as a JSON structure. To support this use case,build a subclass of MutableDict that provides appropriatecoercion to the values placed in the dictionary so that they too are“mutable”, and emit events up to their parent structure.