标记数据帧内容时如何处理NA值？

Question

I have a pandas dataframe and I am trying to tokenize the contents of each row.

import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)

When I run it, I get an error at line 67,

TypeError: ('expected string or buffer', u'occurred at index 67') 

Which I think I am getting because the value for 'Summary' at iloc[67] is an NA value.

TextData.Summary.iloc[67]

Out[45]: nan

Assuming it is the na value which is causing this, is there a way to tell word_tokenize or pandas to ignore the NA values whenever it comes across them?

Else, what else might be causing this?

Answer 1

You can use fillna() to replace NaN with a specified value:

import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData.fillna('some value') # or just: TextData['Summary'].fillna('some value')
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)

---

Previous Answer

You can simply "eliminate" the records where that value is null:

TextData = TextData[TextData['tokenized_summary'].notnull()]

Making the final product look like:

import pandas as pd
import nltk as nk
from nltk import word_tokenize

TextData = pd.read_csv('TextData.csv')
TextData = TextData[TextData['tokenized_summary'].notnull()]
TextData['tokenized_summary'] = TextData.apply(lambda row: nk.word_tokenize(row['Summary']), axis=1)