使用 langdetect 删除非英文文本

Question

I am trying to use langdetect to drop all the languages which are not English in my text.

def det(x):
    try:
        language = detect(x)
    except:
        language = 'Other'
    return language

df['langue'] = df['Tweet'].apply(det)
filtered_for_english = df.loc[df['langue'] == 'en']

The above code is what I have tried. It detects the language used in each tweet but does not drop the non-English tweets from my data frame.

The resulting data frame:

0        es
1        es
2        es
3        en
4        en
         ..
14272    en
14273    en
14274    en
14275    it
14276    en
Name: langue, Length: 14277, dtype: object

How can I fix this code?

Answer 1

This solution worked well for me.

from langdetect import detect

def detect_english(text):
try:
    return detect(text) == 'en'
except:
    return False

Pass the pandas dataframe like the following to eliminate non-English textual data from the dataframe.

df = df[df['text'].apply(detect_english)]

I had 5000 samples and the above implementation removed some and returned 4721 English textual data.

Note: Direct import in colab never worked for me. I had to do !pip install langdetect