如何从字符串中提取标记？

Question

Edit: May someone add a regex solution? I was looking on the following regex:

[\(\)!*-+^]

I had the a function that extracts tokens from a text according to the special chars I declared in its body.

There are 2 problems in the function:

1) It doesn't print the special chars.

2) It outputs wrong when two special chars are next to each other

So I made a change, which fixed problem 1 (As I saw from the result of some tests) But doesn't fix number 2, any help?

Note: I am using C++11 standard and not looking to use boost

Example: Given: a+(b*c) I am expecting: a,+,(,b,*,c,)

Given: a+b I am expecting: a,+,b

Given: ab+ I am expecting: ab,+

Given: a b+ I am expecting: ab,+

Answer 1

You're not handling empty strings well. Before processing the empty string, check to make sure it isn't empty.

#include <sstream>
#include <string>
#include <vector>
#include <cassert>

using std::string;
using std::stringstream;
using std::vector;

namespace {

// Parse a string into a vector of tokens by special characters.
auto find(string str) -> vector<string> {
    auto result = vector<string>{};
    auto pos = string::size_type{};
    auto last_pos = string::size_type{};

    while ((pos = str.find_first_of("+^-*!()", last_pos)) != string::npos) {
        // Is there a token before the special character?
        if (pos - last_pos > 0) {
            result.push_back(str.substr(last_pos, pos - last_pos));
        }

        last_pos = pos + 1;

        // Add the special character as a token.
        result.push_back(str.substr(pos, 1));
    }

    auto last = str.substr(last_pos);

    // Is there a trailing token after the last found special character?    
    if (!last.empty()) {
        result.push_back(str.substr(last_pos));
    }

    return result;
}

// Helper routine.
// Join a vector of strings together using a given separator.
auto join(vector<string> const& v, string sep) -> string {
    auto ss = stringstream{};
    auto first = true;

    for (auto const& s : v) {
        if (first) {
            ss << s;
            first = false;
        } else {
            ss << sep << s;
        }
    }

    return ss.str();
}

// Helper routine.
// Returns a string representing the tokenized string.
auto check(string s) -> string {
    auto v = find(s);
    auto result = join(v, ",");
    return result;
}

} // anon

int main() {
    // Unit tests to check that the string parses and tokenizes into the expected string.
    assert(check("a+(b*c)") == "a,+,(,b,*,c,)");
    assert(check("a+b") == "a,+,b");
    assert(check("ab+") == "ab,+");
    assert(check("a b+") == "a b,+");
    assert(check("a") == "a");
    assert(check("aa") == "aa");
    assert(check("+") == "+");
    assert(check("++") == "+,+");
    assert(check("a+") == "a,+");
    assert(check("+a") == "+,a");
    assert(check("") == "");
}

Answer 2

Here's a regex solution that should parse the tokens you want:

void find(std::string str)
{
    static const std::regex r(R"(\+|\^|-|\*|!|\(|\)|([\w|\s]+))");
    std::copy( std::sregex_token_iterator(str.begin(), str.end(), r, 0),
               std::sregex_token_iterator(),
               std::ostream_iterator<std::string>(std::cout, "\n"));
}

Here's a demo .

Here's an explanation .

Note that this is not a good idea if you want to do general purpose parsing. The regex will quickly become unwieldy (if it's not already), and there are much better tools available to do this for you.