Mahout 0.9为朴素贝叶斯分类文档

Question

I'm very new to Mahout and I am working on classifying unstructured text documents.

I have followed this tutorial as I am using a Naive Bayes Model. I have gotten to the point of training my classifier but I am not sure how to convert a new document into a tfidf vector for classifying.

My data is stored as a TSV file which has a label and the text corresponding to it. I use seq2parse to create the tfidf vectors that is required for training the model.

I then train the model using these tfidf vectors which results in a Naive Bayes model.

Now I have a new unlabelled text document that I wish to classify using this trained model but I am not sure how to convert it into a tfidf vector. If I use seq2parse again then it will create a new set of dictionary file etc and I assume then this doesn't correspond to the dictionary created for the training set.

I have seen a manual implementation of creating the tfidf based on an already created dictionary file and label index at https://github.com/fredang/mahout-naive-bayes-example/blob/master/src/main/java/com/chimpler/example/bayes/Classifier.java but I was wondering if Mahout has already provided some methods to do this just the way they have provided the seq2parse . I would rather use a supporting method of doing it than having to do it manually.

Answer 1

The sample code can help u, perhaps:

org.apache.mahout.math.Vector vector = new RandomAccessSparseVector();
    Integer wordId = dictionary.get(word);  // use hashcode of word

    double tfIdfValue = tfidf.calculate(count, freq.intValue(),
            wordCount, documentCount); // calculate tf*idf

    vector.set(wordId,tfIdfValue);

// Model is a matrix (wordId, labelId) => probability score
NaiveBayesModel model = NaiveBayesModel.materialize(
        new Path(modelPath), configuration);
StandardNaiveBayesClassifier classifier = new StandardNaiveBayesClassifier(
        model);

// With the classifier, we get one score for each label.The label with
// the highest score is the one the tweet is more likely to be
// associated to
Vector resultVector = classifier.classifyFull(vector);