从缓冲区读取记录

Question

I have a huge file to be read. I have a IO thread which reads data ( 4mb ) from the disk and stores in a circular array of 6 elements ( 4mb each ). I have another thread which reads from the circular buffer to convert the data into a some records.

The problem is I can have records which spans across 2 different buffers ( ie. say a record can start from the end of 1st buffer and extend upto start of next buffer )

How do I handle such cases ?

Could you point to some sample implementation ?

Answer 1

Your function for reading from a buffer should read from the next buffer when a record spans two buffers.

More precisely, create a function that assembles a record from data in the buffer. If the data pointer hits the end of a buffer before the record is finished, set the data pointer to the beginning of the next buffer.

Hmmm, looks like this can be applied more generically. Build items by reading from a data pointer. Before the data pointer is accessed, check for end of buffer. If the pointer is past the end of the buffer, set it to the beginning of the next buffer. This concept is very similar to buffered I/O. Hmmm, perhaps you can modify the iostreams or create your own, that will fetch data from your buffers instead of cin or a file. Look at std::istringstream .

Answer 2

You should split your record reading process in to steps :

1. convert your buffers chain into an input stream
2. parse the input stream to produce record

You can use standard classes to achieve the first step as Thomas said or implement your own solution. A trivial solution may look like this (assuming fixed size for records)

class BufferReader{
...
public :
   // this function will read data from buffers. 
   // size of readed data is arbitrary and does not depend on buffer size
   // it will return -1 when eof reached, readed size in other case 
   int readData(char *data, int length);
...
}

Then you can parse your records :

int size = /* size of the record */;
BufferReader br(/* some construction parameters here */)
char data[size];
while(br.readData(data, size) == size){
   // parse your data to fill your record
...