Python share objects between independent processes

Question

I have 3 script files, a.py, b.py and c.py. a.py is used to launch b.py and c.py.

I have an SQL/SQLite/SQLcipher connection in a.py. Im looking for a way to share that connection with b/c.py so each can send a query to a.py and a.py return the result.

I have been looking at sockets and multiprocessing shared_memory but both seem to require a size which can be anything from a few kb to 100+Mb depending on the query.

So is it possible to share a memory address and size so a second python process running in a different shell can access that memory/object?

quick example pseudo code:

b.py -> memory.query("select * from table'") -> a.py
a.py -> q(memory.query): memory.results= cur.exe(memory.query) -> b.py 
b.py -> print(memory.results)

Answer 1

You can use amanager here. However, instead of creating the sql connection in a.py , the connection would be created inside a separate manager process, and access to that object would be allowed through proxies. Here is an example:

from multiprocessing import Process, BaseManager
from c import main_c
from b import main_b


class AccessConnection:
    def __init__(self, connection):
        self.connection = connection

    def get_query(self, query):
        return self.connection.get_query(query)

    # You can include other possible functions you want to allow processes to access

def create_my_sql_connection(*args, **kwargs):
    connection = None  # Initialize your SQL connection
    return AccessConnection(connection)

if __name__ == "__main__":
    BaseManager.register("create_my_sql_connection", create_my_sql_connection)
    manager = BaseManager.start()

    access_connection = manager.create_my_sql_connection()  # Create and store your connection in another process

    # Start a and b
    proc_b = Process(main_b, args=(access_connection, ))
    proc_c = Process(main_c, args=(access_connection,))

Then from inside b and c , you can send and receive queries by doing:

access_connection.get_query("select * from table'")

Keep in mind that all arguments you send to the manager process (via access_connection ), and all values you receive from it, need to be picklable for this to work.