从管道读取Python

Question

I have a process that gathers information from the web and stores into files. I have another process that has the purpose to use this data to do analysis. That is, P1 writes the data to files and P2 used these information to do analysis.

Now my problem is with P2. How can I make P2 get input from the data files as a stream? Eg when the bytes are received from P1 to be also input to P2.

I am thinking of using IPC or something similar to that. I thinked of using PIPES but the problem is that the P1 may block until P2 reads the data. The P1 should never stop receiving data. The P2 can be extremely late. One idea is to use a different thread to send the data to the pipe until there is a time out.

Is this approach correct?

Answer 1

Seems that indeed named pipes are your tool for this.

Very easy to consume as regular files from python or using python's os module