RPyC Memory Service

Often in distributed computing their is a need for sharing state (data) between multiple parallel running processes. RpycMemService allows to host a shared memory service on a network, which can be used to share state (objects) among multiple processes. The RpycMemService is an RPyC service tailored to offer memory management routines. Hosting a service is straight-forward:

from rpyc_mem.service import RpycMemService

rs = RpycMemService('localhost', 18813) # Defaults: host='0.0.0.0', port=random
rs.run()

For maintaining a single snapshot of the memory, RpycMemService is supposed to be ran with a ThreadedServer or variants of it. run() method can take an optional server argument; by default ThreadedServer is used. The kwargs used to run the server can be updated by passing server_kwargs argument.

default_kwargs = {
                    'service': self.__class__,
                    'hostname': self._hostname,  # from __init__
                    'port': self._port,  # from __init__
                    'protocol_config': {
                        'allow_all_attrs': True,
                        'allow_setattr': True,
                        'allow_delattr': True
                    }
                  }

default_kwargs.update(server_kwargs)

Refer to RPyC documentation for the servers that come inbuilt. If the port is left out, a random port is assigned which can be accessed through self.server_obj.port (this is available only after the service is ran).

In the background, RpycMemService maintains a mapping between the key and the object in a dict. All management routines need the unique-key for operating on the corresponding object. The RpycMemService comes with all basic memory management routines; memoize, get, update, delete, is_memoized (remote_import, rpyc_version). Most of them are self explanatory, refer to the API documentation for the specifics of each routine.

There is a concept of remote object and remote object generator in RpycMemService. The term remote signifies that the object is remote to the client machine. The remote machine where the object resides can be any remote machine, but in general we use the remote objects of one service with the same. The client objects should not be used as shared objects (reasons and creating remote objects are explained in the Remote Module guide). The remote object generator is simply a callable which returns a remote object, this is primarily for delayed execution, particularly in the case of memoization (dont need to create a new object if the mapping already exists against the key).