The datastore.core package contains the core parts of datastore, including the base Datastore classes, Key, serializers, collections, shims, etc.
The core package exists mainly because namespace packages cannot include members.
datastore is a generic layer of abstraction for data store and database access. It is a simple API with the aim to enable application development in a datastore-agnostic way, allowing datastores to be swapped seamlessly without changing application code. Thus, one can leverage different datastores with different strengths without committing the application to one datastore throughout its lifetime.
Bases: datastore.core.basic.ShimDatastore
Wraps a datastore with a caching shim optimizes some calls.
Returns whether the object named by key exists. First checks cache_datastore.
Removes the object named by key. Writes to both cache_datastore and child_datastore.
Return the object named by key or None if it does not exist. CacheShimDatastore first checks its cache_datastore.
Stores the object value named by key`self. Writes to both ``cache_datastore` and child_datastore.
Bases: object
A Datastore represents storage for any key-value pair.
Datastores are general enough to be backed by all kinds of different storage: in-memory caches, databases, a remote datastore, flat files on disk, etc.
The general idea is to wrap a more complicated storage facility in a simple, uniform interface, keeping the freedom of using the right tools for the job. In particular, a Datastore can aggregate other datastores in interesting ways, like sharded (to distribute load) or tiered access (caches before databases).
While Datastores should be written general enough to accept all sorts of values, some implementations will undoubtedly have to be specific (e.g. SQL databases where fields should be decomposed into columns), particularly to support queries efficiently.
Returns whether the object named by key exists.
The default implementation pays the cost of a get. Some datastore implementations may optimize this.
Removes the object named by key.
Return the object named by key or None if it does not exist.
None takes the role of default value, so no KeyError exception is raised.
Stores the object value named by key.
How to serialize and store objects is up to the underlying datastore. It is recommended to use simple objects (strings, numbers, lists, dicts).
Returns an iterable of objects matching criteria expressed in query
Implementations of query will be the largest differentiating factor amongst datastores. All datastores must implement query, even using query’s worst case scenario, see :ref:class:`Query` for details.
Bases: datastore.core.basic.ShimDatastore
Represents a collection of datastores.
Appends datastore store to this collection.
Returns the datastore at index.
Inserts datastore store into this collection at index.
Removes datastore store from this collection.
Bases: datastore.core.basic.Datastore
Simple straw-man in-memory datastore backed by nested dicts.
Returns whether the object named by key exists.
Checks for the object in the collection corresponding to key.path.
Removes the object named by key.
Removes the object from the collection corresponding to key.path.
Return the object named by key or None.
Retrieves the object from the collection corresponding to key.path.
Stores the object value named by key.
Stores the object in the collection corresponding to key.path.
Returns an iterable of objects matching criteria expressed in query
Naively applies the query operations on the objects within the namespaced collection corresponding to query.key.path.
Bases: datastore.core.basic.ShimDatastore
Datastore that tracks directory entries, like in a filesystem. All key changes cause changes in a collection-like directory.
For example:
>>> import datastore.core
>>>
>>> dds = datastore.DictDatastore()
>>> rds = datastore.DirectoryDatastore(dds)
>>>
>>> a = datastore.Key('/A')
>>> b = datastore.Key('/A/B')
>>> c = datastore.Key('/A/C')
>>>
>>> rds.get(a)
[]
>>> rds.put(b, 1)
>>> rds.get(b)
1
>>> rds.get(a)
['/A/B']
>>> rds.put(c, 1)
>>> rds.get(c)
1
>>> rds.get(a)
['/A/B', '/A/C']
>>> rds.delete(b)
>>> rds.get(a)
['/A/C']
>>> rds.delete(c)
>>> rds.get(a)
[]
Removes the object named by key. DirectoryDatastore removes the directory entry.
Retrieves directory entries for given key.
Retrieve directory values for given key.
Stores the object value named by `key`self. DirectoryDatastore stores a directory entry.
Returns objects matching criteria expressed in query. DirectoryDatastore uses directory entries.
Bases: datastore.core.basic.Datastore
Represents simple wrapper datastore around an object that, though not a Datastore, implements data storage through a similar interface. For example, memcached and redis both implement a get, set, delete interface.
Removes the object named by key in service.
Return the object in service named by key or None.
Stores the object value named by key in service.
Bases: datastore.core.basic.ShimDatastore
Represents a simple ShimDatastore that applies a transform on all incoming keys. For example:
>>> import datastore.core
>>> def transform(key):
... return key.reverse
...
>>> ds = datastore.DictDatastore()
>>> kt = datastore.KeyTransformDatastore(ds, keytransform=transform)
None
>>> ds.put(datastore.Key('/a/b/c'), 'abc')
>>> ds.get(datastore.Key('/a/b/c'))
'abc'
>>> kt.get(datastore.Key('/a/b/c'))
None
>>> kt.get(datastore.Key('/c/b/a'))
'abc'
>>> ds.get(datastore.Key('/c/b/a'))
None
Returns whether the object named by key is in this datastore.
Removes the object named by keytransform(key).
Return the object named by keytransform(key).
Stores the object names by keytransform(key).
Returns a sequence of objects matching criteria expressed in query
Bases: datastore.core.basic.ShimDatastore
Wraps a datastore with a logging shim.
Returns whether the object named by key exists. LoggingDatastore logs the access.
Removes the object named by key. LoggingDatastore logs the access.
Return the object named by key or None if it does not exist. LoggingDatastore logs the access.
Returns an iterable of objects matching criteria expressed in query. LoggingDatastore logs the access.
Bases: datastore.core.basic.KeyTransformDatastore
Represents a simple ShimDatastore that lowercases all incoming keys. For example:
>>> import datastore.core
>>> ds = datastore.DictDatastore()
>>> ds.put(datastore.Key('hello'), 'world')
>>> ds.put(datastore.Key('HELLO'), 'WORLD')
>>> ds.get(datastore.Key('hello'))
'world'
>>> ds.get(datastore.Key('HELLO'))
'WORLD'
>>> ds.get(datastore.Key('HeLlO'))
None
>>> lds = datastore.LowercaseKeyDatastore(ds)
>>> lds.get(datastore.Key('HeLlO'))
'world'
>>> lds.get(datastore.Key('HeLlO'))
'world'
>>> lds.get(datastore.Key('HeLlO'))
'world'
Returns a lowercased key.
Bases: datastore.core.basic.KeyTransformDatastore
Represents a simple ShimDatastore that namespaces all incoming keys. For example:
>>> import datastore.core
>>>
>>> ds = datastore.DictDatastore()
>>> ds.put(datastore.Key('/a/b'), 'ab')
>>> ds.put(datastore.Key('/c/d'), 'cd')
>>> ds.put(datastore.Key('/a/b/c/d'), 'abcd')
>>>
>>> nd = datastore.NamespaceDatastore('/a/b', ds)
>>> nd.get(datastore.Key('/a/b'))
None
>>> nd.get(datastore.Key('/c/d'))
'abcd'
>>> nd.get(datastore.Key('/a/b/c/d'))
None
>>> nd.put(datastore.Key('/c/d'), 'cd')
>>> ds.get(datastore.Key('/a/b/c/d'))
'cd'
Returns a namespaced key: namespace.child(key).
Bases: datastore.core.basic.KeyTransformDatastore
Represents a simple ShimDatastore that shards/namespaces incoming keys.
Incoming keys are sharded into nested namespaces. The idea is to use the key name to separate into nested namespaces. This is akin to the directory structure that git uses for objects. For example:
>>> import datastore.core
>>>
>>> ds = datastore.DictDatastore()
>>> np = datastore.NestedPathDatastore(ds, depth=3, length=2)
>>>
>>> np.put(datastore.Key('/abcdefghijk'), 1)
>>> np.get(datastore.Key('/abcdefghijk'))
1
>>> ds.get(datastore.Key('/abcdefghijk'))
None
>>> ds.get(datastore.Key('/ab/cd/ef/abcdefghijk'))
1
>>> np.put(datastore.Key('abc'), 2)
>>> np.get(datastore.Key('abc'))
2
>>> ds.get(datastore.Key('/ab/ca/bc/abc'))
2
Returns a nested key.
returns a nested version of basename, using the starting characters. For example:
>>> NestedPathDatastore.nested_path('abcdefghijk', 3, 2)
'ab/cd/ef'
>>> NestedPathDatastore.nested_path('abcdefghijk', 4, 2)
'ab/cd/ef/gh'
>>> NestedPathDatastore.nested_path('abcdefghijk', 3, 4)
'abcd/efgh/ijk'
>>> NestedPathDatastore.nested_path('abcdefghijk', 1, 4)
'abcd'
>>> NestedPathDatastore.nested_path('abcdefghijk', 3, 10)
'abcdefghij/k'
Bases: datastore.core.basic.Datastore
Stores nothing, but conforms to the API. Useful to test with.
Remove the object named by key (does nothing).
Return the object named by key or None if it does not exist (None).
Store the object value named by key (does nothing).
Returns an iterable of objects matching criteria in query (empty).
Bases: datastore.core.basic.DatastoreCollection
Represents a collection of datastore shards.
A datastore is selected based on a sharding function. Sharding functions should take a Key and return an integer.
Returns whether the object is in this datastore.
Removes the object from the corresponding datastore.
Return the object named by key from the corresponding datastore.
Stores the object to the corresponding datastore.
Returns a sequence of objects matching criteria expressed in query
Returns the shard index to handle key, according to sharding fn.
Returns the shard to handle key.
A generator that queries each shard in sequence.
Bases: datastore.core.basic.Datastore
Represents a non-concrete datastore that adds functionality between the client and a lower level datastore. Shim datastores do not actually store data themselves; instead, they delegate storage to an underlying child datastore. The default implementation just passes all calls to the child.
Removes the object named by key.
Default shim implementation simply calls child_datastore.delete(key) Override to provide different functionality.
Return the object named by key or None if it does not exist.
Default shim implementation simply returns child_datastore.get(key) Override to provide different functionality, for example:
def get(self, key):
value = self.child_datastore.get(key)
return json.loads(value)
Stores the object value named by key.
Default shim implementation simply calls child_datastore.put(key, value) Override to provide different functionality, for example:
def put(self, key, value):
value = json.dumps(value)
self.child_datastore.put(key, value)
Returns an iterable of objects matching criteria expressed in query.
Default shim implementation simply returns child_datastore.query(query) Override to provide different functionality, for example:
def query(self, query):
cursor = self.child_datastore.query(query)
cursor._iterable = deserialized(cursor._iterable)
return cursor
Bases: datastore.core.basic.ShimDatastore
Datastore that creates filesystem-like symbolic link keys.
A symbolic link key is a way of naming the same value with multiple keys.
For example:
>>> import datastore.core
>>>
>>> dds = datastore.DictDatastore()
>>> sds = datastore.SymlinkDatastore(dds)
>>>
>>> a = datastore.Key('/A')
>>> b = datastore.Key('/B')
>>>
>>> sds.put(a, 1)
>>> sds.get(a)
1
>>> sds.link(a, b)
>>> sds.get(b)
1
>>> sds.put(b, 2)
>>> sds.get(b)
2
>>> sds.get(a)
2
>>> sds.delete(a)
>>> sds.get(a)
None
>>> sds.get(b)
None
>>> sds.put(a, 3)
>>> sds.get(a)
3
>>> sds.get(b)
3
>>> sds.delete(b)
>>> sds.get(b)
None
>>> sds.get(a)
3
Creates a symbolic link key pointing from target_key to source_key
Stores the object named by key. Follows links.
Returns objects matching criteria expressed in query. Follows links.
Bases: datastore.core.basic.DatastoreCollection
Represents a hierarchical collection of datastores.
Each datastore is queried in order. This is helpful to organize access order in terms of speed (i.e. read caches first).
Datastores should be arranged in order of completeness, with the most complete datastore last, as it will handle query calls.
Returns whether the object is in this datastore.
Removes the object from all underlying datastores.
Return the object named by key. Checks each datastore in order.
Stores the object in all underlying datastores.
Returns a sequence of objects matching criteria expressed in query. The last datastore will handle all query calls, as it has a (if not the only) complete record of all objects.
Bases: object
A Key represents the unique identifier of an object.
Our Key scheme is inspired by file systems and the Google App Engine key model.
Keys are meant to be unique across a system. Keys are hierarchical, incorporating more and more specific namespaces. Thus keys can be deemed ‘children’ or ‘ancestors’ of other keys:
Key('/Comedy')
Key('/Comedy/MontyPython')
Also, every namespace can be parametrized to embed relevant object information. For example, the Key name (most specific namespace) could include the object type:
Key('/Comedy/MontyPython/Actor:JohnCleese')
Key('/Comedy/MontyPython/Sketch:CheeseShop')
Key('/Comedy/MontyPython/Sketch:CheeseShop/Character:Mousebender')
Returns the child Key by appending namespace other.
>>> Key('/Comedy/MontyPython').child('Actor:JohnCleese')
Key('/Comedy/MontyPython/Actor:JohnCleese')
Returns an instance Key, by appending a name to the namespace.
Returns whether this Key is an ancestor of other.
>>> john = Key('/Comedy/MontyPython/Actor:JohnCleese')
>>> Key('/Comedy').isAncestorOf(john)
True
Returns whether this Key is a descendant of other.
>>> Key('/Comedy/MontyPython').isDescendantOf(Key('/Comedy'))
True
Returns whether this Key is top-level (one namespace).
Returns the list representation of this Key.
Note that this method assumes the key is immutable.
Returns the name of this Key, the value of the last namespace.
Returns the list of namespaces of this Key.
Returns the parent Key (all namespaces except the last).
>>> Key('/Comedy/MontyPython/Actor:JohnCleese').parent
Key('/Comedy/MontyPython')
Returns the path of this Key, the parent and the type.
Returns a random Key
Returns the path string path without duplicate slashes.
Returns the reverse of this Key.
>>> Key('/Comedy/MontyPython/Actor:JohnCleese').reverse
Key('/Actor:JohnCleese/MontyPython/Comedy')
Returns the type of this Key, the field of the last namespace.
Bases: str
A Key Namespace is a string identifier.
A namespace can optionally include a field (delimited by ‘:’)
Example namespaces:
Namespace('Bruces')
Namespace('Song:PhilosopherSong')
returns the field part of this namespace, if any.
returns the value part of this namespace.
Bases: object
Represents a query result generator.
Naively apply query filters.
Naively apply query limit.
Naively apply query offset.
Naively apply query orders.
Iterator next. Build up count of returned elements during iteration.
Bases: object
Represents a Filter for a specific field and its value.
Filters are used on queries to narrow down the set of matching objects.
field: the attribute name (string) on which to apply the filter.
value: the attribute value to compare against.
Examples:
Filter('name', '=', 'John Cleese')
Filter('age', '>=', 18)
Conditional operators that Filters support.
Returns the elements in iterable that pass given filters
Generator function that iteratively filters given items.
Object attribute getter. Can be overridden to match client data model. See datastore.query._object_getattr().
Returns whether this value passes this filter
Bases: object
Represents an Order upon a specific field, and a direction. Orders are used on queries to define how they operate on objects
Examples:
Order('+name') # ascending order by name
Order('-age') # descending order by age
Order('score') # ascending order by score
A key function to be used in pythonic sort operations.
Returns a function that will compare two items according to orders
Object attribute getter. Can be overridden to match client data model. See datastore.query._object_getattr().
Ordering operators: + is ascending, - is descending.
Returns the elements in items sorted according to orders
Bases: object
A Query describes a set of objects.
Queries are used to retrieve objects and instances matching a set of criteria from Datastores. Query objects themselves are simply descriptions, the actual Query implementations are left up to the Datastores.
Returns a copy of this query.
Returns a dictionary representing this query.
Adds a Filter to this query.
Returns self for JS-like method chaining:
query.filter('age', '>', 18).filter('sex', '=', 'Female')
Constructs a query from a dictionary.
Attribute getter for the objects to operate on.
This function can be overridden in classes or instances of Query, Filter, and Order. Thus, a custom function to extract values to attributes can be specified, and the system can remain agnostic to the client’s data model, without loosing query power.
For example, the default implementation works with attributes and items:
def _object_getattr(obj, field):
# check whether this key is an attribute
if hasattr(obj, field):
value = getattr(obj, field)
# if not, perhaps it is an item (raw dicts, etc)
elif field in obj:
value = obj[field]
# return whatever we've got.
return value
Or consider a more complex, application-specific structure:
def _object_getattr(version, field):
if field in ['key', 'committed', 'created', 'hash']:
return getattr(version, field)
else:
return version.attributes[field]['value']
Adds an Order to this query.
Returns self for JS-like method chaining:
query.order('+age').order('-home')
A generator that chains iterables.
A generator that applies a count limit.
A generator that applies an offset, skipping offset elements from iterable. If skip_signal is a callable, it will be called with every skipped element.
Bases: datastore.core.serialize.Serializer
Implements serializing protocol but does not serialize at all. If only storing strings (or already-serialized values).
returns value.
returns value.
Bases: object
Serializing protocol. Serialized data must be a string.
returns serialized value.
returns deserialized value.
Bases: datastore.core.basic.ShimDatastore
Represents a Datastore that serializes and deserializes values.
As data is put, the serializer shim serializes it and put``s it into the underlying ``child_datastore. Correspondingly, on the way out (through get or query) the data is retrieved from the child_datastore and deserialized.
datastore: a child datastore for the ShimDatastore superclass.
serializer: a serializer object (responds to loads and dumps).
Returns deserialized value or None.
Return the object named by key or None if it does not exist. Retrieves the value from the child_datastore, and de-serializes it on the way out.
Stores the object value named by key. Serializes values on the way in, and stores the serialized data into the child_datastore.
Returns an iterable of objects matching criteria expressed in query De-serializes values on the way out, using a deserialized_gen to avoid incurring the cost of de-serializing all data at once, or ever, if iteration over results does not finish (subject to order generator constraint).
Returns serialized value or None.
Bases: datastore.core.serialize.Serializer, list
represents a stack of serializers, applying each serializer in sequence.
returns serialized value.
Returns deserialized value.
Generator that yields deserialized objects from iterable.
Bases: datastore.core.serialize.Serializer
map serializer that ensures the serialized value is a mapping type.
returns mapping typed serialized value.
Returns mapping type deserialized value.
Patch bson in pymongo to use loads and dumps interface.
Bases: datastore.core.serialize.Serializer
json wrapper serializer that pretty-prints. Useful for human readable values and versioning.
returns json serialized value (pretty-printed).
returns json deserialized value.
Generator that yields serialized objects from iterable.
Return a SerializerShimDatastore wrapping datastore.
Can be used as a syntacticly-nicer eay to wrap a datastore with a serializer:
my_store = datastore.serialize.shim(my_store, json)