用C ++解析巨大的csv文件

Question

I order to simulate my network I am using a trace file (csv file) with a size between 5 to 30 GB. The csv file is a row based, where each row contains multiple fields delimited by a space and forming teh information to form a network packet:

3     53      4    12    1     1  2  6

Since the file's size could reach several GBs (millions of lines), is it better to divided it into small chunks myfile00.csv, myfile01.csv..., or I can process the entire file on the hard drive without being loaded into the memory? I want to read the file line by line at a specific time, which is the clock cycle of the simulation, and get all information in the line to create an omnet++ message.

packet MyTrace::getpacket() {
int id; // first field
int cycle; // second field
int source; // third field
int destination; // fourth field
int numberofDep; // fifth field
std::list<int> listofDep; // remaining fields

if (!traceFile.is_open()) {
 // get id
 // get cycle
 // ....
}

Any suggestion would be helpful.

EDIT:

  string line;
  ifstream myfile ("BlackSmall.csv");
    int currentline=0 ;
  if (myfile.is_open())
  {
   while (getline(myfile, line)) {  

      istringstream ss(line);
      string request;
      int id, cycle, source , dest, srcType, destType, packetSize, dependency;
      int listdep;
      std::list<int> dep; 
              ss >> id; 
              ss>> cycle; 
              ss>> source; 
              ss>> dest;
              ss>>request;
              ss>> srcType;
              ss>> destType;
              ss>> packetSize;
              ss>> dependency; 
              while (ss >> listdep) dep.push_back(listdep);
           // Create my packet

    }
    myfile.close();
    }
  else cout << "Unable to open file"; 

With the above code, I can get all information that I need from a line. The problem is that I need to use this code inside a class, which when I call it returns just one line's information. Is there a way how to point to a specific line when I call this class?

Answer 1

It seems like your application seems to require a single sequential pass through the input, so processing a file that is 1GB or 100GB is perhaps just a matter of patience and perhaps parallelism.

The approach should be to translate records line-by-line. You should avoid strategies that attempt to read the entire file into memory. The STL offers the easy-to-use std::ifstream class with a built-in getline method, which returns a std::string containing the line to be converted.

If you are feeling more ambitious and want to control the amount of data read or buffered more carefully then you would not be the first developer to roll-your-own code to implement a buffered reader. This is a fairly empowering exercise and will help you think through some corner cases with reading partial lines and such. But in the end, it probably will not give you a significant boost toward your goal. I suspect the ifstream approach will get you up and running without the hassle and will not ultimately be the bottleneck in processing these files.

If you were really concerned about optimizing execution time then having multiple files might help you launch parallel processing tasks.

// define a class to hold your custom record
class Record {
};

// create a parser function to convert a line of text into the record
bool parse(std::string const &line, Record &record) {
}

// create a translator method to convert a record into the desired output
bool write(Record const &record, std::ofstream &os) {
}

// actually open input stream for the input file
std::ifstream is;
std::ofstream os;
std::string line;

while (std::getline(is,line)) {
  Record record;
  if (!parse(line,record)) break;
  if (!write(record,os)) break;
}

You can re-use the Record instance by moving it outside the while loop so long as you are careful to reset the variable so that information from preceding records does not taint the current record. You can also dive head first into the C++ ecosystem by producing stream input and output operator ("<<",">>") but I personally find this approach to be more confusion than it is worth.

Answer 2

Perhaps best approach for you would be to import your CSV file into SQLite database.

Once you import it and add some indexes, you can easily and very efficiently query necessary rows from that database. SQLite has lots of ready-to-use C/C++ client libraries available, you can start with default one at https://www.sqlite.org/cintro.html .