如何在 CSV 文件中找到单词的频率？

Question

Total beginner here. I have a CSV file of around 20k tweets that I scraped (about a specific topic). I want to get a list of the most popular hashtags and the most tagged accounts. When I tried looking for a solution online, most answers have code in which the words they want to find the frequency of are already known (ie find the frequency of the word "apple" in a string). However, I don't know what the most popular hashtag/tagged account is in this CSV and I want to grab a list of the top few.

Basically what I want to do:

1. For each tweet (row in the CSV), find if "#" (or "@") appears.
2. If a # or @ appears, grab the word/phrase following the "#" or "@".
3. Count the frequency of words/phrases that follow the #s or @s.
4. Sort list of #s or @s that appear most frequently in a tweet's content.

For example, if the tweets were all about Baseball, maybe most of the tweets would have #mlb or tag @Yankees.

Most popular hashtags:

1. baseball
2. mlb
3. sports

Most tagged accounts:

1. @mlb
2. @baseball
3. @yankees

Answer 1

One option I can think of is to use a dictionary. I would read in my CSV and then split it on spaces so I get individual words. Then check the first character of the word for # or @ and depending on which one it is I would use their respective dictionaries. If it's a # then check if the word already exists in the # dictionary and if it does then increment its value by 1, otherwise add it and give it a value of 1 (same goes for the @ dictionary).

Answer 2

Here's sample code that outputs json file with counts of each hashtagh and tag found in the tweets. Split into hashtag and tag counts.

import pandas as pd
import re
import json


df = pd.read_csv("data.csv")
tweets = df["Tweet"]

counts = {"hashtags": [], "tags": []}
for tweet in tweets:
    hashtags = re.findall("(#\w+)", tweet)
    for hashtag in hashtags:
        if any(hashtag in item["text"] for item in counts["hashtags"]):
            for item in counts["hashtags"]:
                if item["text"] == hashtag:
                    item["count"] += 1
        else:
            counts["hashtags"].append({"text": hashtag, "count": 1})
    
    tags = re.findall("(@\w+)", tweet)
    for tag in tags:
        if any(tag in item["text"] for item in counts["tags"]):
            for item in counts["tags"]:
                if item["text"] == tag:
                    item["count"] += 1
        else:
            counts["tags"].append({"text": tag, "count": 1})


counts["hashtags"] = sorted(counts["hashtags"], key=lambda x: x["count"], reverse=True)
counts["tags"] = sorted(counts["tags"], key=lambda x: x["count"], reverse=True)


with open("counts.json", "w") as f:
    json.dump(counts, f, indent=4, sort_keys=True)

Sample data used

Tweet
This is tweet 1 #life #thug @tweeter asdasd
This is tweet 2 #life #thug @tweeter qweqwe
This is tweet 3 #life#thug @tweeter asdasd @tweetking
This is tweet 4 #liferocks #life
This is tweet 5 #liferocks #earth
This is tweet 6 #liferocks #storm
This is tweet 7 #liferocks #fire
This is tweet 8 #nothing

Answer 3

No need to use a dictionary or go through each to do a tally or count. Pandas can do the count and sums by column.

There's 2 ways you could do it given the data set Tsingis useds:

import pandas as pd

df = pd.DataFrame([
'This is tweet 1 #life #thug @tweeter asdasd',
'This is tweet 2 #life #thug @tweeter qweqwe',
'This is tweet 3 #life#thug @tweeter asdasd @tweetking',
'This is tweet 4 #liferocks #life',
'This is tweet 5 #liferocks #earth',
'This is tweet 6 #liferocks #storm',
'This is tweet 7 #liferocks #fire',
'This is tweet 8 #nothing'], columns=['Tweet'])

Now if we do:

print(df['Tweet'].str.count("#liferocks"))
0    0
1    0
2    0
3    1
4    1
5    1
6    1
7    0

It'll count how many times it shows up in each row. So it's just a matter of taking the .sum()

print(df['Tweet'].str.count("#liferocks").sum())
4

However, slight problem with this is if we do:

print(df['Tweet'].str.count("#life"))
0    1
1    1
2    1
3    2
4    1
5    1
6    1
7    0

Notice row index 3, we have a count of it twice, because it sees '#life 2 times with '#life' and '#liferocks' .

So we'll slightly modify the search/count to find the full hastags, meaning start at the hashtag and the specific word, and keep looking until you reach a space or another #. Then count those.

Code:

import pandas as pd

df = pd.read_csv("data.csv")
tweets = df["Tweet"]

def count_exact_words(df, column_name, word):
    df['Tweet'] = df['Tweet'] + '.'
    return word, df['Tweet'].str.count(r'({word})[\s#.]'.format(word=word)).sum()

print(count_exact_words(df, 'Tweet', '#liferocks'))
print(count_exact_words(df, 'Tweet', '#life')) 

Output:

('#liferocks', 4)
('#life', 4)