Python threading — How to know if thread already is running?

Question

I have a thread I use to process data. Right now it triggers every time it detects a new file in a folder. I am coding in Python, but maybe it is more of a general programming question?

My question is two-fold:

1. Should I use a trigger like that (event-driven, more or less), or should I be using time based (every 3 minutes, create a new thread)?

2. If I go with time-based and create a new thread, wouldn't it cause problems if the two threads are processing the same data? Is there a way to tell them to work together or to not spawn a second one if one exists?

I apologize for the probable naivety of my question, I am still quite new to multi-threading and mutliple processes, so I still don't know when to use what.

Answer 1

As far as I understand you process any new file with a separate thread, so it behaves like a server processing multiple requests with a single routine.

1) I think that time-triggered creation isn't good in your case because it doesn't depend on either system performance or number of files to process. You may run a few threads as daemons and have a main thread that assign tasks to these threads as soon as they come. If there are too many at the same time, you just drop new tasks. On the other hand, you may create a new thread that does processing each time new file appears and then join it when the processing has finished

2) You may start new thread explicitly giving it the file name. Whether it's possible or not for a few threads to work with a single file simultaneously would depend on what you exactly do with the file. In general it becomes way more complicated than single file per thread