Using QTimer for Streaming Large Data File in Python

Question

I'll start with saying that I have working code...but that doesn't make it ideal code which is why I want to run my approach by the community. I'm trying to do this the "QT way" which is forcing me down roads I don't normally go.

I have some large 400+mb binary files containing raw time variant data. I need to plot this data to the user so that the data playback matches the time duration of the recording. I have a working approach using a QTimer.timeout to trigger a file read. I read x amount of bytes, and when the read is complete I emit a signal to trigger the plotting operation. By adjusting my timeout duration I can control the rate of plotting without blocking my interface(not blocking the GUI is key). This seems to work, but it feels overly complicated for something as simple as a file.read.

When I receive data in a stream over TCP I can use the socket.readReady signal to tell me when to process data. Since the data is arriving serially in time, it naturally looks right over the TCP stream.

I have essentially duplicated the readReady of a socket by using fread and emitting a signal. Does this sound like a reasonable approach?

Answer 1

An alternative might be a plotting buffer being somewhat bigger than what you need to display. If it runs short of values you fill it up. If you do this in a thread you can ensure data availability without the need of timers or read ready signals. You would just have to keep track of which data has already been read and the direction in time.

Answer 2

An alternative could be using the QFile::map() function to map the right slice of data to display into memory for direct access without any file reading. I guess that should be fast enough for fast displaying depending on the slice size. This approach may be combined with the buffer approach above to avoid excessive mapping. This would mean to map a slice larger that currently needed.