Word frequency by iterating over a list of dictionaries - python

Question

In reference to a previous question, I am hoping to iterate over a list of dictionaries and turn the output into a new data frame. For now, I have a CSV with two columns. A column with a word and another with a URL (see below).

| Keyword  | URL                     | 
| -------- | --------------          |
| word1    | www.example.com/topic-1 |
| word2    | www.example.com/topic-2 |
| word3    | www.example.com/topic-3 |
| word4    | www.example.com/topic-4 |

I have turned this CSV into a list of dictionaries and am attempting to iterate over those lists to get a count of how often the word shows on the URL.

My code can be seen in this colab notebook .

My hope is to have a final output that looks like this:

| Keyword | URL                        | Count |
|:----    |:------:                    | -----:|
| word1   | www.example.com/topic-1    | 1003  |
| word2   | www.example.com/topic-2    | 405   |
| word3   | www.example.com/topic-3    | 123   |
| word4   | www.example.com/topic-4    | 554   |

The 'Count' column being the frequency of 'word1' on 'www.example.com/topic-1'.

Any help is appreciated!

Answer 1

Using DataFrame.apply to create a new column using a function of the other columns:

import pandas as pd
import requests

df = pd.DataFrame({'Keyword': ['code', 'apply', 'midnight'],
                   'URL': ['https://stackoverflow.com/questions/70581444/word-frequency-by-iterating-over-a-list-of-dictionaries-python/',
                           'https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html',
                           'https://stackoverflow.com/questions/62694219/minimum-number-of-platforms-required-for-a-railway-station'
                          ]})

print(df)
#     Keyword                                                URL
# 0      code  https://stackoverflow.com/questions/70581444/w...
# 1     apply  https://pandas.pydata.org/docs/reference/api/p...
# 2  midnight  https://stackoverflow.com/questions/62694219/m...



def get_count(row):
    r = requests.get(row['URL'], allow_redirects=False)
    count = r.text.lower().count(row['Keyword'].lower())
    return count

df['Count'] = df.apply(get_count, axis=1)

print(df)
#     Keyword                                                URL  Count
# 0      code  https://stackoverflow.com/questions/70581444/w...     32
# 1     apply  https://pandas.pydata.org/docs/reference/api/p...     32
# 2  midnight  https://stackoverflow.com/questions/62694219/m...     18