我们如何创建标记的语料库

Question

I am trying to create my own tagged corpus on demonetization dataset and the dataset has approximately 6250 tweets. the code is below, although it is giving the results for a small dataset of size 200 entries.

df = pd.read_csv('Demonetization_data29th2.csv',encoding = "ISO-8859-1")

text = df['CONTENT']
sentiment = df['sentiment']
a =[]

tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]
tagged = [nltk.pos_tag(sent) for sent in tagged]
print tagged[0]

print "---------"

brown_tagged_sents = tagged
print brown_tagged_sents[0]
size = int(len(brown_tagged_sents) * 0.7)
tags = [tag for (word, tag) in brown.tagged_words()]
defaultTag = nltk.FreqDist(tags).max()
print defaultTag
train_sents = brown_tagged_sents[:size]
test_sents = brown_tagged_sents[size:]
tagger = ngramTagger(train_sents, 2, defaultTag)
print tagger.evaluate(test_sents)

i got an error like :

File "C:/Users/HP/pos-2.py", line 41, in <module>
    tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 130, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 97, in sent_tokenize
    return tokenizer.tokenize(text)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1235, in tokenize
    return list(self.sentences_from_text(text, realign_boundaries))

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1283, in sentences_from_text
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1274, in span_tokenize
    return [(sl.start, sl.stop) for sl in slices]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1314, in _realign_boundaries
    for sl1, sl2 in _pair_iter(slices):

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 312, in _pair_iter
    prev = next(it)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1287, in _slices_from_text
    for match in self._lang_vars.period_context_re().finditer(text):

TypeError: expected string or buffer

Please guide me where I am going wrong

Answer 1

you probably got a null in one or more rows of that column.

fill it with empty string like:

df['CONTENT'] = df['CONTENT'].fillna('')