Check if existing process exists - if it does, communicate with it, otherwise create a new one

Question

I'm sure that this question has been asked before, but I couldn't find it.

I have written a Python program which, given a directory, uses a very simple heuristic to determine how to "unpack" the content of it and to what target directory.

This program is executed whenever a new download has completed. If a lot of downloads complete at around the same, I get a lot of processes concurrently unpacking stuff. I'd like to fix this problem by rewriting large portions of the program to only unpack one directory at a time.

So to achieve this, I figured I'd use a "lock/PID" file which contains the PID of any currently executing program. If the lock/PID file exists, the newly spawn processes should simply send something along the lines of ("queue", "D:/some/directory") to the existing process and have that process unpack that target when it's done with its current unpacking.

How would I achieve this in Python? This must work on Windows systems, but ideally on GNU/Linux as well.

Answer 1

if you just want to check if the PID file exists, you can use: os.path

os.path.exists(path) 

Answer 2

So since you're already using something like the lockfile by Ben Finney

example:

 from lockfile.pidlockfile import PIDLockFile
 lock = PIDLockFile('somefile')
 lock.acquire(timeout=3600)
 #do stuff
 lock.release()

You might want to communicate with a running daemon you should have the daemon listen to some socket, then from a spawned process send data to the socket. (f.ex a udp socket)

so in the daemon:

import socket
import traceback
import Queue
import threading
import sys

hostname = 'localhost'
port = 12368
#your lockfile magick here
# if you want one script to run each time, put client code here instead of seperate script
#if already running somehwere:
    #message = "hello"
    #sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM )
    #sock.sendto( message, (hostname, port) )
    #sys.exit(0)

que = Queue.Queue(0)

socket_ = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
socket_.bind((hostname, port))

class MyThread(threading.Thread):
    def run(self):
        while True: #maybe add some timeout here, so this process closes after a while
                    # but only stop if que.empty()
            message = que.get()
            print "handling message: %s" % message
            #handle message

t = MyThread()
t.start()

while True:
        try:
            #blocking call
            message, _ = socket_.recvfrom(8192) #buffer size
            print "received message: %s"  % message
            #queue message
            que.put(message)
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            traceback.print_exc()

on the client:

import socket
hostname = 'localhost'
port = 12368
message = "hello"
sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM )
sock.sendto( message, (hostname, port) )

you can use 'localhost' for the hostname if you're running all of this on the same machine.

On the other hand, using the multiprocess pipes instead of sockets might be the proper way, but I've got no experience in them yet. This setup has the added benefit of being able to run the server and client on a different machine.