如何从开头和结尾删除空格？ 分词器（split='[.??]'）

Question

tokenizer = Tokenizer(split='[.!?]') #create a token based on sentences
tokenizer.fit_on_texts(df['Cleaned'].values)
X_data = tokenizer.texts_to_sequences(df['Cleaned'].values)
X_sequ = pad_sequences(X_data)

I got the list of tokens whith whitespace, like blow #let see the frist 10 of our text sequences

list(tokenizer.word_index)[:10]

Output like this below

[' rnfbdhl yis',
 ' oromoon bilisoomsiteeti jirti',
 ' namni oromummaa isaatiin mataa gadi qabtee deemu hin jiru yeroo ammaa tanatti']

how i can remove whitespace from start and end automatically? please help?

Answer 1

You can try regular expression using re in python. So here it goes, The caret symbol tells us the beginning of a string, the dollar is the end of string, \s+ means one or more spaces. So the regular expression means, replace every space(one or more) at the beginning or end of a string with ''(nothing). Let me know if it has worked for you.

import re
[re.sub(r'^\s+|\s+$', '', item) for item in list(tokenizer.word_index)]

With given string:

x = [' rnfbdhl yis', ' oromoon bilisoomsiteeti jirti', ' namni oromummaa isaatiin mataa gadi qabtee deemu hin jiru yeroo ammaa tanatti']

Output :

[re.sub(r'^\s+|\s+$', '', item) for item in x]
['rnfbdhl yis', 'oromoon bilisoomsiteeti jirti', 'namni oromummaa isaatiin mataa gadi qabtee deemu hin jiru yeroo ammaa tanatti']