Reading data from hard drive and put it in a c++ container

Question

I wrote a simple thread and its task is just to read the data from hard drive , put it in a container and tag it with timestamp and an unique Id. After that I will write the newly structured data into a memory mapped file.

The thing is I don't care about the internal structure of the data, I mean it can be in a Wav format (because in the real situtaion I will be dealing with some audio datas which are avergely 3 mb each) bu I won't do any operaions on that data. After inserting it in my struct I will be just dealing with the UniqueId and Data Tag. Sample structure would be something like:

Struct SampleData 
{
long UniqueID;
...  MyData; // the data which I am trying to read from hard drive
Time  insertionTime;
}

So the question is how am i going to read a Wav data into this struct without knowing (because i don't need to) the internal structure of it? What will ve the ... part for instance. Is there any container type for a large data chunk?

For reading tha data may I use ifstream or any other method?

Answer 1

Try to keep it in a TLV format:
http://en.wikipedia.org/wiki/Type-length-value

EDIT: A very simple container for TLV.

You'll be able to store the raw data as it is and you'll know which field you're reading and what's its size.

class TlvContainer
{
public:
  unsigned long Type; // Maybe we have billions of types of objects?
  unsigned long Size; // The size of the object.
  unsigned char* Bytes; // This will hold the raw data.
};

When you write your data to the file, you'll have to know how many bytes it is, allocate the "Bytes" array and update the "Size" field.
When you read it from the file, you'll know how you have written it in. (You'll have to read the fields in the same order you have written them.)

For example, if you wrote it as: Type, Size, Bytes:
You'll first read sizeof(unsigned long) from the file to know the the type of the element.
Then you'll need to read another sizeof(unsigned long) to know how big is your real data.
and then you'll be able to read "Size" bytes from the file, knowing that after them there is a new element built the same way.

Answer 2

How about storing MyData as vector<unsigned char> ?

You can use file streams to read but remember to use ios::binary mode. See http://www.cplusplus.com/doc/tutorial/files/

Here is sample code. You may want to add error checking. Also, I didn't try to compile this so there may be mistakes.

std::vector<unsigned char> data;
ifstream file("sample.wav", ios::binary);
while(!file.eof()) {
    unsigned char byte;
    file >> byte;
    data.push_back(byte);
 }