Hadoop Mapreduce CSV作为关键字：word

Question

I couldn't find the answer to my question, if there is a similar post please refer me there.

I have a CSV file that I am trying to perform a mapreduce on, the format of the CSV is two columns: Book Title | Synopsis. I want to be able to perform a mapreduce on each book and have a count for the words in each book, thus, I would like the output to be: Book Title : Token.

So far, I have attempted to use the following code to achieve this:

    String firstBook = null;
    while (itr.hasMoreTokens()) {
        String secondBook = itr.nextToken();
        if (firstBook != null) {
              word.set(firstBook + ":" + secondBook);
              context.write(word, one);
        }
        firstBook = secondBook;
      } 

This sometimes outputs the following; word : title

In addition, it limits the analysis I can do as this is the logic I would like to use to perform an analysis of bigrams in each synopsis.

Is there a way that I can isolate each book title, just performing the mapreduce on the 'synopsis' column of the CSV? If so, how would I do this and obtain the desired output?

Many thanks in advance.

UPDATE

The code is modified from Hadoops wordcount example, the only change is in the "map" section and is shown above. You can find the input data here .

Representation of the CSV File:

Book title, Synopsis
A short history of nearly everything, Bill Byrson describes himself as a reluctant traveller...
Reclaiming economic development, There is no alternative to neoliberal economics - or so it appeared...

-> Note I have shortened the synopsis.

Answer 1

thus, I would like the output to be: Book Title : Token.

If you copied the word count example, you're only writing every two tokens followed by the number 1. It doesn't look like you're taking the titles, only the tokens of the synopsis. But you've cut off the part where you get a tokenizer, so it's hard to tell.

Note: If a book title contained commas, you'll end up with part of the title as part of the synopsis with your current approach. If possible, you should make the title column quoted, or better, not use commas (or any other common delimiter) between the columns if that delimiter is going to be part of at least the first column.

perform an analysis of bigrams in each synopsis.

If you want to do that type of analysis, I would recommend you clean up the columns first - remove capitalization and punctuation. Stemming of words might also produce better output.

Is there a way that I can isolate each book title

Sure, put an if statement for the first column targeting a specific book, and only write to the context in that condition

Otherwise, if your mapper would write the book title only as the key, then they will be isolated as part of the reduce function

Answer 2

This was solved by using the "KeyValueTextInputFormat" class, there are several tutorials on here that specifically relate to this class. This allowed me to separate the CSV file, resulting in a key : value pair (in my case, book title : synopsis). You can then perform a reduce as normal on the "value" and pass this through to the reduce stage as "key : token".