C ++-在通过tokenizer读取文件行时删除或跳过引号字符

Question

I have a csv file that has records like:

837478739*"EP" 1 "3FB2B464BD5003B55CA6065E8E040A2A"*"F"*21*15*"NH"*"N" 0 *-1*"-1"*0*0**-1*223944*-1*"23" 1 "-1" "-1" "78909" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "-1" "74425" "26" "-1"*"-1"*1*1*69*23.58*0*0*0*0*"MC"

The file has lots of records, so I need a fast method to breakdown the line and push_back each of those parts into a vector. The main reason I choose tokenizer is that I heard a lot about its performance. I have a function:

void break(){
   //using namespace boost;
   string s = "This is a , test '' file";
   boost::tokenizer<> tok(s);
   vector<string> line;
   for(boost::tokenizer<>::iterator beg=tok.begin();beg!=tok.end();++beg){
       line.push_back(*beg);
   }
   cout << line[3] << "  and  " << line[5] << endl;
}

By that I can get each part of the sentence and ignore everything that is not a letter. Does the tokenizer have the ability to read the record that I have and parse them by "*" delimiter and remove the quotes from the string? There won't be any kind of special character between quotes, I just need to remove the quote marks. I tried to read the tokenizer document, but nothing came out.

Answer 1

You can use regex_replace .

"break" is keyword. You shouldn't use it for function name.

Answer 2

You need to assign another TokenizerFunc to your Tokenizer to parse the string differently, the default parses on space and punctuation

http://www.boost.org/doc/libs/1_37_0/libs/tokenizer/tokenizerfunction.htm