计算文本文件中每个单词的出现次数

Question

给定一个具有多个字符串的大型文本文件，在C ++中读取文本文件并计算每个单词有多少次出现的最有效方法是什么？ 文本文件的大小未知，所以我不能只使用简单的数组。 此外，还有另一个问题。 该文本文件的每一行都以类别关键字开头，随后的单词是该类别的功能。 我需要能够计算出该类别中每个单词的出现次数。

例如：

colors red blue green yellow orange purple
sky blue high clouds air empty vast big
ocean wet water aquatic blue
colors brown black blue white blue blue

在此示例中，我需要计算一下，在“ 颜色 ”类别中，尽管总共总共出现了6次蓝色，但还是出现了4次“ 蓝色 ”。

Answer 1

我将使用流来读取和分隔单词（它通过查找空格来分隔单词）并将它们保存到字典中 （标准的C ++方法是使用std::map ）。

这是C ++记录的代码：

#include <iostream>
#include <map> // A map will be used to count the words.
#include <fstream> // Will be used to read from a file.
#include <string> // The map's key value.
using namespace std;


//Will be used to print the map later.
template <class KTy, class Ty>
void PrintMap(map<KTy, Ty> map)
{
    typedef std::map<KTy, Ty>::iterator iterator;
    for (iterator p = map.begin(); p != map.end(); p++)
        cout << p->first << ": " << p->second << endl;
}

int main(void)
{
    static const char* fileName = "C:\\MyFile.txt";

    // Will store the word and count.
    map<string, unsigned int> wordsCount;


    {
        // Begin reading from file:
        ifstream fileStream(fileName);

        // Check if we've opened the file (as we should have).
        if (fileStream.is_open())
            while (fileStream.good())
            {
                // Store the next word in the file in a local variable.
                string word;
                fileStream >> word;

                //Look if it's already there.
                if (wordsCount.find(word) == wordsCount.end()) // Then we've encountered the word for a first time.
                    wordsCount[word] = 1; // Initialize it to 1.
                else // Then we've already seen it before..
                    wordsCount[word]++; // Just increment it.
            }
        else  // We couldn't open the file. Report the error in the error stream.
        {
            cerr << "Couldn't open the file." << endl;
            return EXIT_FAILURE;
        }

        // Print the words map.
        PrintMap(wordsCount);
    }

    return EXIT_SUCCESS;
}

输出：

空气：1
水生的：1
大：1
黑色：1
蓝色：6
棕色：1
云：1
颜色：2
空：1
绿色：1
高：1
海洋：1
橙色：1
紫色：1
红色：1
天空：1
广泛：1
水：1
湿的：1
白色：1
黄色：1

Answer 2

标记单词并将它们存储为键值对。

更新：我意识到我读错了问题。 以下代码应分开并按类别进行计数：

#include <iostream>
#include <string>
#include <map>
#include <fstream>
using namespace std;
int main()
{
    ifstream file;
    file.open("path\\to\\text\\file");
    if(!file.is_open()) return 1;
    map<string, map<string, int> > categories;
    while(file.good())
    {
        string s;
        getline(file, s);
        int pos = s.find_first_of(' ');
        if(pos < 0) continue;
        string word = s.substr(0, pos);
        string category = word;
        s = s.erase(0, pos+1);
        while(s.size() > 0)
        {
            pos = s.find_first_of(' ');
            if(pos < 0)
                pos = s.size();
            string word = s.substr(0, pos);
            if(word != "")
                categories[category][word]++;
            s = s.erase(0, pos+1);
        }
    }
    for(map<string, map<string, int> >::iterator cit = categories.begin(); cit != categories.end(); ++cit)
    {
        cout << "Category - " << cit->first << endl;
        for(map<string, int>::iterator wit = cit->second.begin(); wit != cit->second.end(); ++wit)
            cout << "\tword: " << wit->first << ",\t" << wit->second << endl;
    }
    return 0;
}

更新2：克里斯要求对算法进行修改：

#include <iostream>
#include <string>
#include <map>
#include <fstream>
using namespace std;
int main()
{
    ifstream file;
    file.open("D:\\Documents\\txt.txt");
    if(!file.is_open()) return 1;
    map<string, int> categories;
    while(file.good())
    {
        string s;
        getline(file, s);
        int pos = s.find_first_of(' ');
        if(pos < 0) continue;
        while(s.size() > 0)
        {
            pos = s.find_first_of(' ');
            if(pos < 0)
                pos = s.size();
            string word = s.substr(0, pos);
            if(word != "")
                categories[word]++;
            s = s.erase(0, pos+1);
        }
    }
    for(map<string, int>::iterator wit = categories.begin(); wit != categories.end(); ++wit)
        cout << "word: " << wit->first << "\t" << wit->second << endl;
    return 0;
}

Answer 3

这是一个可以实现您指定目标的解决方案。 看到它住在这里 。

它利用std::map来维护（类别，单词）对出现次数的计数。

std::istringstream用于首先将数据分成行，然后分成字。

---

输出：

(colors, black) => 1
(colors, blue) => 4
(colors, brown) => 1
(colors, green) => 1
(colors, orange) => 1
(colors, purple) => 1
(colors, red) => 1
(colors, white) => 1
(colors, yellow) => 1
(ocean, aquatic) => 1
(ocean, blue) => 1
(ocean, water) => 1
(ocean, wet) => 1
(sky, air) => 1
(sky, big) => 1
(sky, blue) => 1
(sky, clouds) => 1
(sky, empty) => 1
(sky, high) => 1
(sky, vast) => 1

程序：

#include <iostream>  // std::cout, std::endl
#include <map>       // std::map
#include <sstream>   // std::istringstream
#include <utility>   // std::pair

int main()
{
    // The data.
    std::string content =
        "colors red blue green yellow orange purple\n"
        "sky blue high clouds air empty vast big\n"
        "ocean wet water aquatic blue\n"
        "colors brown black blue white blue blue\n";

    // Load the data into an in-memory table.
    std::istringstream table(content);

    std::string row;
    std::string category;
    std::string word;
    const char delim = ' ';
    std::map<pair<std::string, std::string>, long> category_map;
    std::pair<std::string, std::string> cw_pair;
    long count;

    // Read each row from the in-memory table.
    while (!table.eof())
    {
        // Get a row of data.
        getline(table, row);

        // Allow the row to be read word-by-word.
        std::istringstream words(row);

        // Get the first word in the row; it is the category.
        getline(words, category, delim);

        // Get the remaining words in the row.
        while (std::getline(words, word, delim)) {
            cw_pair = std::make_pair(category, word);

            // Maintain a count of each time a (category, word) pair occurs.
            if (category_map.count(cw_pair) > 0) {
                category_map[cw_pair] += 1;
            } else {
                category_map[cw_pair] = 1;
            }
        }
    }

   // Print out each unique (category, word) pair and
   // the number of times that it occurs.
   std::map<pair<std::string, std::string>, long>::iterator it;

   for (it = category_map.begin(); it != category_map.end(); ++it) {
       cw_pair = it->first;
       category = cw_pair.first;
       word = cw_pair.second;
       count = it->second;

       std::cout << "(" << category << ", " << word << ") => "
           << count << std::endl;
   }
}