使用 langdetect 计算 Pandas 数据框中的语言频率

Question

I want to find the frequency of different languages in a tweet dataset. I eventually only want to use the tweets that are in English but want to find out the frequencies of other languages as well.

I've detected the language of tweets within my dataset using langdetect, and now I want to count the frequency of each language. This is my code for detecting the language:

from langdetect import detect    
import pandas as pd
data_path = "./output_1.csv"
df =  pd.read_csv(data_path, index_col=0)

for index, row in df.iterrows():
    print(detect(row['text']))
    if detect(row['text']) == "en":
        print(row['text'])

I wanted to use list property count to count the frequencies:

using the list i = ['en','fr','es','it','cs','pt']
d = {x:i.count(x) for x in i}
print d

How do I use the count property on the data that I got using langdetect?

Answer 1

To create a separate column containing the language you could do:

df['language'] = df['text'].apply(lambda x: detect(x))

Then to count the frequency you could do:

pd.DataFrame(df.groupby('language').text.count().sort_values(ascending=False))