从 C++ 中的字符串中提取标记

Question

Edit: I'm looking for a solution that doesn't use regex since it seems buggy and not trustable

I had the following function which extracts tokens of a string whenever one the following symbols is found: +,-,^,*,!

bool extract_tokens(string expression, std::vector<string> &tokens) {    
    static const std::regex reg(R"(\+|\^|-|\*|!|\(|\)|([\w|\s]+))");
    std::copy(std::sregex_token_iterator(right_token.begin(), right_token.end(), reg, 0),
              std::sregex_token_iterator(),
              std::back_inserter(tokens));
    return true;
}

I though it worked perfectly until today I found an edge case, The following input: !aaa + ! a is supposed to return ,,aaa,+,!, a But it returns ,,aaa,+,"",!, a Notice the extra empty string between + and..

How may I prevent this behaviour? I think this can be done with the regex expression,

Answer 1

Inspired by https://stackoverflow.com/a/9436872/4645334 you could solve the problem with:

bool extract_tokens(std::string expression, std::vector<std::string> &tokens) {
  std::string token;

  for (const auto& c: expression) {
    if (c == '/' || c == '-' || c == '*' || c == '+' || c == '!') {
      if (token.length() && !std::all_of(token.cbegin(), token.cend(), [](auto c) { return c == ' '; })) tokens.push_back(token);
      token.clear();
      tokens.emplace_back(1, c);
    } else {
      token += c;
    }
  }

  if (token.length() && !std::all_of(token.cbegin(), token.cend(), [](auto c) { return c == ' '; })) tokens.push_back(token);
     
  return true;
}

Input:

"!aaa + ! a"

Output:

"!","aaa ","+","!"," a"

Answer 2

In an attempt to salvage the regular expression-based solution, I came up with this:

[-+^*!()]|\s*[^-+^*!()\s][^-+^*!()]*

Demo . This reports delimiters, and anything between delimiters including leading and trailing whitespace, but drops tokens consisting of whitespace alone.

---

A similar expression that also strips leading and trailing whitespace:

[-+^*!()]|[^-+^*!()\s]+(\s+[^-+^*!()\s]+)*)

Demo