数据清理：如何从pandas数据框列中删除某些值？

Question

I am working on the analysis of the user profile interest of a social network. I have generated a dataframe with User id, Name and User interest from the export of the social network database. I was supposed to only get keywords in the 'User interest' column. but actually, I got a mix of keywords and User ID...

    User ID displayName interests
0   5705952d0eb2063205ca1d3c    Jane Catch  []
1   5705e99ac391580e00ea87c9    Heidi Kent  [{u'text': u'psychology', u'_id': {u'$oid': u'...
2   5705efb6c391580e00ea87ca    Rob Tuckinson   [{u'text': u'learning', u'_id': {u'$oid': u'57...

I would like to make some data cleaning on the column interests to only keep the keywords in the user interest column.

Today, I have this information:

User ID,displayName,interests
"570df0f2a40cc20e00c15e09,Alejandra Zara,""[{u'text': u'pretend-play', u'_id': {u'$oid': u'570e57eba40cc20e00c161ea'}}, {u'text': u'autobiographical-memory', u'_id': {u'$oid': u'570e57eba40cc20e00c161e9'}}]"""

For the first line, I would like only to keep the information below:

"570df0f2a40cc20e00c15e09,Alejandra Zara,pretend-play', autobiographical-memory'

Any ideas of data cleaning techniques? Each time, I need to remove the information relative to user ID (different for each row such as:

u'_id': {u'$oid': u'570e57eba40cc20e00c161ea'}}

and remove {u'text': u (which is placed at the beginning of each keyword).

Answer 1

If I'm reading the question correctly, what you have in your interests column is the string representation of a Python list of dict s from which you want to get specific values. If so, you can use ast.literal_eval to parse it:

In [24]: df
Out[24]: 
                    User ID     displayName  \
0  570df0f2a40cc20e00c15e09  Alejandra Zara   

                                           interests  
0  [{u'text': u'pretend-play', u'_id': {u'$oid': ...  

In [25]: df['interests'].map(lambda x: ','.join(i['text'] for i in ast.literal_eval(x)))
Out[25]: 
0    pretend-play,autobiographical-memory
Name: interests, dtype: object