boost::split leaves empty tokens at the beginning and end of string - is this desired behaviour?

Question

Since I couldn't find anything on this in the documentation , I thought I ask it here. I have the following program (C++11):

#include <iostream> 
#include <boost/algorithm/string.hpp>

using namespace std;
using namespace boost;

int main () {
    string tmp = " #tag #tag1#tag2  #tag3 ####tag4   ";
    list<iterator_range<string::iterator> > matches;
    split( matches, tmp, is_any_of("\t #"), token_compress_on );

    for( auto match: matches ) {
            cout << "'" << match << "'\n";
    }
}

The output is:

''
'tag'
'tag1'
'tag2'
'tag3'
'tag4'
''

I would have thought that the token_compress_on option removes all empty tokens. The solution is, for example, to use boost::trim_if . Nevertheless I was wondering if this is the desired behaviour of boost::split, and why this is happening?

(g++ 4.6.3, boost 1.48)

Answer 1

The behavior is intentional, because you could recreate the string (complete with starting and trailing spaces) from the split version. Boost doesn't know if that whitespace is significant or not to you (it might be, as some file formats, for example, might force leading spaces/specific space counts).

You should trim_if or trim as you are if you do need to remove leading/trailing spaces.

Answer 2

If eCompress argument is set to token_compress_on , adjacent separators are merged together. Otherwise, every two separators delimit a token.

Here

It does not remove the tokens only merges them.

Answer 3

boost::split always returns n + 1 tokens where n is number of separators in the input string. So don't be surprised when it returns 1 token when you pass it an empty string.

The rationale behind it is quite simple. Imagine that you are parsing a CSV file. You need to get the exactly same number of elements regardless whether the last token is empty or not.

It is much easier to remove the empty tokens than to guess whether they were supposed to be in the result or not. Credit

This behaviour is similar to python

>>> len("".split(','))
1