Multiprocesses python with shared memory

Question

I have an object that connects to a websocket remote server. I need to do a parallel process at the same time. However, I don't want to create a new connection to the server. Since threads are the easier way to do this, this is what I have been using so far. However, I have been getting a huge latency because of GIL. Can I achieve the same thing as threads but with multiprocesses in parallel?

This is the code that I have:

class WebSocketApp(object):

  def on_open(self):

    # Create another thread to make sure the commands are always been read
    print "Creating thread..."
    try: 
        thread.start_new_thread( self.read_commands,() )
    except:
        print "Error: Unable to start thread"

Is there an equivalent way to do this with multiprocesses?

Thanks!

Answer 1

You sure can, use something along the lines of:

from multiprocessing import Process

class WebSocketApp(object):

  def on_open(self):

    # Create another thread to make sure the commands are always been read
    print "Creating thread..."
    try: 
        p = Process(target = WebSocketApp.read_commands, args = (self, )) # Add other arguments to this tuple
        p.start()
    except:
        print "Error: Unable to start thread"

It is important to note, however, that as soon as the object is sent to the other process the two objects self and self in the different threads diverge and represent different objects. If you wish to communicate you will need to use something like the included Queue or Pipe in the multiprocessing module.

You may need to keep a reference of all the processes ( p in this case) in your main thread in order to be able to communicate that your program is terminating (As a still-running child process will appear to hang the parent when it dies), but that depends on the nature of your program.

If you wish to keep the object the same, you can do one of a few things:

Make all of your object properties either single values or arrays and then do something similar to this:

from multiprocessing import Process, Value, Array

class WebSocketApp(object):
  def __init__(self):
      self.my_value = Value('d', 0.3)
      self.my_array = Array('i', [4 10 4])

  # -- Snip --

And then these values should work as shared memory. The types are very restrictive though (You must specify their types) A different answer is to use a manager:

from multiprocessing import Process, Manager

class WebSocketApp(object):
  def __init__(self):
      self.my_manager = Manager()
      self.my_list = self.my_manager.list()
      self.my_dict = self.my_manager.dict()

  # -- Snip --

And then self.my_list and self.my_dict act as a shared-memory list and dictionary respectively.

However, the types for both of these approaches can be restrictive so you may have to roll your own technique with a Queue and a Semaphore . But it depends what you're doing.

Check out the multiprocessing documentation for more information.

Answer 2

The direct equivalent is

import multiprocessing

class WebSocketApp(object):

  def on_open(self):

    # Create another process to make sure the commands are always been read
    print "Creating process..."
    try: 
        multiprocessing.Process(target=self.read_commands,).start()
    except:
        print "Error: Unable to start process"

However, this doesn't address the "shared memory" aspect, which has to be handled a little differently than it is with threads, where you can just use global variables. You haven't really specified what objects you need to share between processes, so it's hard to say exactly what approach you should take. The multiprocessing documentation does cover ways to deal with shared state , however. Do note that in general it's better to avoid shared state if possible , and just explicitly pass state between the processes, either as an argument to the Process constructor or via a something like a Queue .