Extract hashtags from entities and add them to new column

Question

I created the following dataframe from a json tweets file.

tweets['text'] =    list(map(lambda tweet: tweet['text'], tweets_data))
tweets['date'] =    list(map(lambda tweet: tweet['created_at'], tweets_data))
tweets['lang'] =    list(map(lambda tweet: tweet['lang'], tweets_data))
tweets['place_type'] = list(map(lambda tweet: tweet['place']['place_type'] if tweet['place'] != None else None, tweets_data))
tweets['place_name'] = list(map(lambda tweet: tweet['place']['full_name'] if tweet['place'] != None else None, tweets_data))
tweets['country'] = list(map(lambda tweet: tweet['place']['country'] if tweet['place'] != None else None, tweets_data))
tweets['hashtags'] =  list(map(lambda tweet: tweet['entities']['hashtags'], tweets_data))

The current output of the hashtags column is the following:

"[{'text': 'NOJUSTICE', 'indices': [62, 72]}, {'text': 'TrumpPence', 'indices': [93, 104]}]"

I'm trying to create a new column that only contains the hashtags. For example:

NOJUSTICE and Trumpence

I tried the following code:

ht=[]
for s in range(len(tweets['hashtags'])):
    hasht=[]
    for t in range(len(tweets.hashtags[s])):
        hasht.append(tweets['hashtags'][s][t]['text'])
        t=t+1
    ht.append(hasht)
    s=s+1
tweets['HT']=zip(ht)

and I received the following error:

TypeError                                 Traceback (most recent call last)
<ipython-input-38-9780008354d7> in <module>
      3     hasht=[]
      4     for t in range(len(tweets.hashtags[s])):
----> 5         hasht.append(tweets['hashtags'][s][t]['text'])
      6         t=t+1
      7     ht.append(hasht)

TypeError: string indices must be integers

sample csv file

i want the hashtags column to only contain the hashtags without the indices and other stuff

UPDATE: i used the following suggestion and got the same error

tweets['hashtags'].apply(lambda x: [d['text'] for d in x])

Answer 1

You can use a pandas.Series.apply to parse the hashtag as follows:

tweets['hashtags'].apply(lambda x: [d['text'] for d in x])

Then you need to store the output in a new column of the dataframe. This anonymous function takes the value as input and iterates over each dictionary in the list (if it's non empty), then creates a new list of the values in key: 'text'. The output would look like this:

0                                                  []
1                             [NOJUSTICE, TrumpPence]
2                                                  []
3                                                  []
4                          [CNN, newday, Trump, ISIS]
5                                        [NeverTrump]
6    [Trump2016, TrumpPence16, MakeAmericaGreatAgain]
7                                                  []
8                                                  []
9                                                  []
dtype: object