c++ specify dividers for reading words from text file

Question

I have the following code prints each unique word and its count from a text file (contains >= 30k words), however it's separating words by whitespace, I had results like so:

how can I modify the code to specify the expected dividers?

template <class KTy, class Ty>
void PrintMap(map<KTy, Ty> map)
{
    typedef std::map<KTy, Ty>::iterator iterator;
    for (iterator p = map.begin(); p != map.end(); p++)
        cout << p->first << ": " << p->second << endl;
}

void UniqueWords(string fileName) {
    // Will store the word and count.
    map<string, unsigned int> wordsCount;

    // Begin reading from file:
    ifstream fileStream(fileName);

    // Check if we've opened the file (as we should have).
    if (fileStream.is_open())
        while (fileStream.good())
        {
            // Store the next word in the file in a local variable.
            string word;
            fileStream >> word;

            //Look if it's already there.
            if (wordsCount.find(word) == wordsCount.end()) // Then we've encountered the word for a first time.
                wordsCount[word] = 1; // Initialize it to 1.
            else // Then we've already seen it before..
                wordsCount[word]++; // Just increment it.
        }
    else  // We couldn't open the file. Report the error in the error stream.
    {
        cerr << "Couldn't open the file." << endl;
    }

    // Print the words map.
    PrintMap(wordsCount);
}

Answer 1

You can use a stream with a std::ctype<char> facet imbue() ed which considers whatever characters you fancy as space. Doing so would look something like this:

#include<locale>
#include<cctype>

struct myctype_table {
    std::ctype_base::mask table[std::ctype<char>::table_size];
    myctype_table(char const* spaces) {
        while (*spaces) {
            table[static_cast<unsigned char>(*spaces)] = std::ctype_base::isspace;
        }
    }
};
class myctype
    : private myctype_table,
    , public std::ctype<char> {
public:
    myctype(char const* spaces)
        : myctype_table(spaces)
        , std::ctype<char>(table) {
    };
};

int main() {
     std::locale myloc(std::locale(), new myctype(" \t\n\r?:.,!"));
     std::cin.imbue(myloc);
     for (std::string word; std::cin >> word; ) {
         // words are separated by the extended list of spaces
     }
}

This code isn't test right now - I'm typing on a mobile device. I probably misused some of the std::cypte<char> interfaces but something along those lines after fixing the names, etc. should work.

Answer 2

As you expect the forbidden characters at the end of the found word you can remove them prior to push the word into wordsCount:

if(word[word.length()-1] == ';' || word[word.length()-1] == ',' || ....){
   word.erase(word.length()-1);
}

Answer 3

After fileStream >> word; , you can call this function. Take a look and see if it's clear:

string adapt(string word) {
    string forbidden = "!?,.[];";
    string ret = "";
    for(int i = 0; i < word.size(); i++) {
        bool ok = true;
        for(int j = 0; j < forbidden.size(); j++) {
            if(word[i] == forbidden[j]) {
                ok = false;
                break;
            }
        }
        if(ok)
            ret.push_back(word[i]);
    }
    return ret;
}

Something like this:

fileStream >> word;
word = adapt(word);