Backends
mellea.backends.cache
Module: mellea.backends.cache
Caching strategies.
Classes
class mellea.backends.cache.Cache()
A Cache for storing model state (e.g., kv cache).
Methods
mellea.backends.cache.Cache.put(key: str, value: Any)
Inserts into the cache. May result in eviction of other cached values.
mellea.backends.cache.Cache.get(key: str)
Retrieves a value from the cache. Returns None if the id has no cached value. May impact which cache values are evicted.
mellea.backends.cache.Cache.current_size()
Returns the number of things currently in the cache. Mostly useful for debugging.
class mellea.backends.cache.SimpleLRUCache(capacity: int)
A simple LRU cache.
Constructor
Initializes the LRU cache with a certain capacity. TheSimpleLRUCache either contains a value or it doesn’t. There is no cache hierarchy. Take care when choosing capacity. In practice usually a small value will be fine, but ideally you should try to choose a capacity based upon your available device memory and the context size of your model.