mt.base.object_store
In-memory object store that can be used by multiple processes.
MT is tring to make sure that the functions in this module are multiprocessing-safe and thread-safe.
Functions
create(): Creates an in-memory object store.valid(): Checks whether an object is a valid object store.count(): Returns the number of objects in the store.keys(): Gets the list of keys the store contains.has(): Checks if an object exists the store given its key.get(): Gets an object in the store given its key.put(): Puts an object in the store.remove(): Attempts to remove an object in the store based on its key.
- mt.base.object_store.create(min_free_mem_pct=0.2, put_policy='rotate')
Creates an in-memory object store.
- Parameters:
min_free_mem_pct (float) – the minimum percentage of free physical memory over total physical memory where an object can be put to the store without restriction. When the free memory percentage drops below the given value, a put policy is activated to decide how to proceed further
put_policy ({'rotate', 'strict'}) – policy for putting an object to the object store when the free memory percentage drops below the given value in min_free_mem_pct. If ‘rotate’ is given, the object store keeps removing earliest-accessed-time objects from the store until enough memory is available or the store is empty. If ‘strict’ is given, it does nothing. Then, it stores the object if there is enough memory.
- Returns:
an object store that can be passed to other processes.
- Return type:
multiprocessing.managers.DictProxy
- mt.base.object_store.valid(store)
Checks whether an object is a valid object store.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
- Returns:
whether or not the input argument is a valid object store
- Return type:
bool
- mt.base.object_store.count(store)
Returns the number of objects in the store.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
- Returns:
number of objects in the store
- Return type:
int
- mt.base.object_store.keys(store)
Gets the list of keys the store contains.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
- Returns:
list of keys
- Return type:
list
- mt.base.object_store.has(store, key)
Checks if an object exists the store given its key.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
key (str) – the key to identify the object
- Returns:
whether or not the key exists in the store
- Return type:
bool
- mt.base.object_store.get(store, key, default_value=None)
Gets an object in the store given its key.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
key (str) – the key to identify the object
- Returns:
the object associated with the key, or default value if not found
- Return type:
object
- mt.base.object_store.put(store, key, value)
Puts an object in the store.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
key (str) – the key to identify the object
value (object) – the object itself
- Returns:
whether or not the object has been stored. Check
create()for more details about the put policy.- Return type:
bool
- Raises:
TimeoutError – if we cannot lock the object store to proceed
- mt.base.object_store.remove(store, key)
Attempts to remove an object in the store based on its key.
- Parameters:
store (multiprocessing.managers.DictProxy) – object store
key (str) – the key to identify the object
- Returns:
whether or not the object has been removed.
- Return type:
bool
- Raises:
TimeoutError – if we cannot lock the object store to proceed