RPyC Memory

By default all the objects created through RemoteModule are unnamed. Once the reference is lost the object can not be accessed anymore. In a distributed setup we need a definitive way to access or share the object across the processes. For this we map the remote object against an unique-key. RpycMem exactly does that, it wraps around the remote object acting as a proxy while creating or fetching the existing remote object against the mapping. RpycMem needs an RpycMemConnect object for performing the sync. The same connection object can be re-purposed for both RemoteModule and RpycMem instead of creating a new connection.

from multiprocessing import Process

from rpyc_mem.connect import RpycMemConnect
from rpyc_mem.client import RemoteModule, RpycMem


def target_proc(unique_key):
    print('Child Process:')

    rc = RpycMemConnect('localhost', 18813)
    rp = RemoteModule(rc)

    # Generators allow delayed execution (dont create if mapping already exists)
    rm = RpycMem(rc, unique_key, robj_gen=lambda: rp().list([1, 2, 3]))
    print(rm)
    rm.append(3)

    print('---------')


print('Parent Process:')

# Assuming service is running on localhost:18813
rc = RpycMemConnect('localhost', 18813)
rp = RemoteModule(rc)

rm = RpycMem(rc, 'unique-key', rp().list([1, 2])) # Either pass robj or robj_gen

print(rm)
print(len(rm))  # Python special method lookup

proc = Process(target=target_proc, args=('unique-key',))
proc.start()
proc.join()

print(rm)

print('---------')

"""
Output:
Parent Process:
[1, 2]
2
Child Process:
[1, 2]
---------
[1, 2, 3]
---------
"""

Till now we are able to create the objects of builtins/modules that remote has to offer. But often times we need to share the objects of custom classes. RemoteModule has no direct way to create such objects since the remote is not aware of your class declarations. The way around is to share the attributes (class attributes, in most cases) of the class instead of the entire class. In the following code, the attributes lock and obj of Shared class are shared instead of the entire class

import multiprocessing
from multiprocessing import Process

from rpyc_mem.connect import RpycMemConnect
from rpyc_mem.client import RemoteModule, RpycMem

# Assuming service is running on localhost:18813
rc = RpycMemConnect('localhost', 18813)
rp = RemoteModule(rc)


class Shared:
    lock = RpycMem(rc, 'lock-key', robj_gen=lambda: rp('threading').Lock())
    obj = RpycMem(rc, 'obj-key', robj_gen=lambda: rp().list([1, 2]))

    def __init__(self, title):
        self.title = title

    def describe(self):
        with self.lock:
            print('%s: %s' % (self.title, self.obj))

    @classmethod
    def run(cls):
        with cls.lock:
            if isinstance(cls.obj, list):
                print(cls.obj)
                cls.obj.rmem_update(rp().tuple([1, 2]))
            else:
                print('Oops')


if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')  # Refer https://github.com/tomerfiliba-org/rpyc/issues/482

    proc1 = Process(target=Shared.run)
    proc2 = Process(target=Shared.run)
    proc3 = Process(target=Shared('Cool-Class').describe)

    proc1.start()
    proc2.start()
    proc3.start()

    proc1.join()
    proc2.join()
    proc3.join()

"""
Output (varied):
[1, 2]
Cool-Class: (1, 2)
Oops
"""

Setting the multiprocessing start method to spawn is important on Unix based systems (On Windows and MacOs spawn is the default start method) because this causes the file to be reimported, which will create a fresh RPyC connection. Otherwise two processes will talk to the server from a similar socket object (socket address), which will compromise the data integrity on the server side. However, one can rely on RpycMemSession and forget about the connections being reused in different processes. Refer to RPyC Memory Session guide for more information.

The object proxying idea is inspired from the Tomer Filiba’s Python recipe. The proxy objects of RpycMem class will behave like the original objects in most of the cases. However, they come with few limitations. Consider the below interactive session:

>>> from rpyc_mem.connect import RpycMemConnect
>>> from rpyc_mem.client import RpycMem

>>> rc = RpycMemConnect('localhost', 18813)

>>> rm = RpycMem(rc, 'key', 1)
>>> rm = rm + 1 # rm variable is replaced by int and the proxy object is garbage collected
>>> print(rm)
2
>>> print(type(rm))
<class 'int'>
>>> rm = RpycMem(rc, 'key1', 1)
>>> _ = rm.rmem_update(rm + 1)  # Use rmem_update to update the shared memory object.
>>> print(rm)
2
>>> print(type(rm))
<class 'rpyc_mem.client.rpyc_mem_object.RpycMem'>