C++ 读取 txt 的随机行？

Question

I am running C++ code where I need to import data from txt file. The text file contains 10,000 lines. Each line contains n columns of binary data.

The code has to loop 100,000 times, each time it has to randomly select a line out of the txt file and assign the binary values in the columns to some variables.

What is the most efficient way to write this code? should I load the file first into the memory or should I randomly open a random line number?

How can I implement this in C++?

Answer 1

To randomly access a line in a text file, all lines need to have the same byte-length. If you don't have that, you need to loop until you get at the correct line. Since this will be pretty slow for so much access, better just load it into a std::vector of std::string s, each entry being one line (this is easily done with std::getline ). Or since you want to assign values from the different columns, you can use a std::vector with your own struct like

struct MyValues{
  double d;
  int i;
  // whatever you have / need
};

std::vector<MyValues> vec;

Which might be better instead of parsing the line all the time.

With the std::vector , you get your random access and only have to loop once through the whole file.

Answer 2

10K lines is a pretty small file. If you have, say, 100 chars per line, it will use the HUGE amount of 1MB of your RAM.

Load it to a vector and access it the way you want.

Answer 3

maybe not THE most efficient, but you could try this:

int main() {
    //use ifstream to read
    ifstream in("yourfile.txt");

    //string to store the line
    string line = "";

    //random number generator
    srand(time(NULL));

    for(int i = 0; i < 100000; i++) {
        in.seekg(rand() % 10000);
        in>>line;
        //do what you want with the line here...
    }
}

Im too lazy right now, but you need to make sure that you check your ifstream for errors like end-of-file, index-out-of-bounds, etc...

Answer 4

Since you're taking 100,000 samples from just 10,000 lines, the majority of lines will be sampled. Read the entire file into an array data structure, and then randomly sample the array. This avoids file seeking entirely.

The more common case is to sample only a small subset of the file's data. To do that, assuming the lines are different length, seek to random points in the file, skip to the next newline (for example cin.ignore( numeric_limits< streamsize >::max(), '\n' ) , and then parse the subsequent text.