使用正则表达式标记 C++ 中的 stream

Question

I would like to tokenize a stream in C++ using a regular expression similar to the way this is done for a string:

std::vector<std::string> tokenize(const std::string& source, const std::regex& re)
{
    auto tokens = std::vector<std::string>(
        std::sregex_token_iterator{ begin(source), end(source), re, -1 },
        std::sregex_token_iterator{}
    );
    return tokens;
}

The difference would be passing an istream as the source variable.

I could first copy the contents of the stream into a string, and then perform the tokenization, but this seems to be inefficient.

Answer 1

transform_reduce(istream_iterator<string>(cin), istream_iterator<string>(),
               vector<string>{},
               [](auto&& a, auto&& b) {
                 auto acc = [](auto&& h, auto&& w) { h.emplace_back(w); return move(h); };
                 if constexpr (is_same_v<decay_t<decltype(a)>, vector<string>>)
                   return accumulate(istream_iterator<string>(b), istream_iterator<string>(), move(a), acc);
                 else
                   return accumulate(istream_iterator<string>(a), istream_iterator<string>(), move(b), acc);
               },
               [&](auto l) { replace_if(l.begin(), l.end(), is_word_seperators, ' '); return stringstream{move(l)}; }
             );