在python中，如何将两列的CSV文件转换成bigrams？

Question

I'd like to turn the given csv file into bigrams:

demo.csv:

words   class
hi my name is Jeff. brown
Wow, I am awesome.  red
I am a professional.    red
Will you marry me?  red
How are you today?  brown
Today, I woke up with a smile on my face.   red
My day today has been amazing.  brown

Answer 1

First, make sure to read your data and select the word-column. You can use pandas.read_csv for that. Since I dont have your.csv-file, I have recreated the data like that:

import pandas as pd
df = pd.DataFrame(
    ["hi my name is Jeff.",
     "Wow, I am awesome.",
     "I am a professional.",
     "Will you marry me?",
     "How are you today?",
     "Today, I woke up with a smile on my face.",
     "My day today has been amazing."], columns=['words'])

which looks like this:

                                       words
0                        hi my name is Jeff.
1                         Wow, I am awesome.
2                       I am a professional.
3                         Will you marry me?
4                         How are you today?
5  Today, I woke up with a smile on my face.
6             My day today has been amazing.

A library you can use to create bigrams is nltk . In this example I create a function that returns the bigrams as a list.

import nltk
def bigrams(words):
    return list(nltk.bigrams(nltk.word_tokenize(words)))

And then apply this function to my DataFrame and assign the result to a new column called bigrams like this:

df["bigrams"] = df.words.apply(bigrams)

the new column now looks like this:

0    [(hi, my), (my, name), (name, is), (is, Jeff),...
1    [(Wow, ,), (,, I), (I, am), (am, awesome), (aw...
2    [(I, am), (am, a), (a, professional), (profess...
3    [(Will, you), (you, marry), (marry, me), (me, ?)]
4    [(How, are), (are, you), (you, today), (today,...
5    [(Today, ,), (,, I), (I, woke), (woke, up), (u...
6    [(My, day), (day, today), (today, has), (has, ...

I hope this helps, feel free to ask any question or tell me if you want to change something:)