提取并替换字符之间的字符串

Question

I have cleaned my data which was encoded in 'utf-8' . Using .str.extract() , I got the text enclosed between [(u'text')] to 'text format however, my code is not registering garbage/unicode characters "\\u09xx" and similar type of literals. How should I remove them ?

Input :

{"HT" : ["([u'SoccerTips', u'FootballTips'],)", "([u'\u092b\u094c\u091c\u0940', u'FixedMatch', u'CT2017Final'],)"]}

My code :

df1 = df.drop('HT', axis=1).join(
             df.HT
             .str
             .split(expand=True)
             .stack()
             .reset_index(drop=True, level=1)
             .rename('HT')           
             )

df1['HT'] = df1['HT'].str.extract("u+(\'[^\']*)", expand=False).fillna('')
df1['HT'] = "#" + df1['HT']

Output :-

{"HT" : ["#'SoccerTips" , "#'FootballTips", "#'\u092b\u094c\u091c\u0940", "#'FixedMatch", "#'CT2017Final"]}

Expected Output :-

{"HT" : ["#SoccerTips" , "#FootballTips", " ", "#FixedMatch", "#CT2017Final"]}

Answer 1

A possible solution:

import pandas as pd

# the input
df1= {"HT" : ["([u'SoccerTips', u'FootballTips'],)", "([u'\u092b\u094c\u091c\u0940', u'FixedMatch', u'CT2017Final'],)"]}

# convert to Dataframe
df1= pd.DataFrame(df1)

# cleaning
df1.HT.replace('\(\[|\],\)','', regex=True, inplace=True)
df1.HT.replace("u'[^\x00-\x7f]*'","", regex=True, inplace=True)
df1.HT.replace("u'([^\']+)'",'#\\1', regex=True, inplace= True)
df1.HT= df1.HT.str.split(', ')

# final result
df1= {'HT':[j for i in df1.HT for j in i]}

# output: df1 -> {'HT': ['#SoccerTips', '#FootballTips', '', '#FixedMatch', '#CT2017Final']}