阅读大字符串并拆分每个单词在python中花费太多时间

Question

I am reading a dataframe column having comments. The data is taking forever to read using the code below. Is there a way to make this faster ?

for val in df.Description:
    val = str(val)
    tokens = val.split()  
    for i in range(len(tokens)):
        tokens[i] = tokens[i].lower()  
        for words in tokens:
            comment = comment + words + ''

df.Description is a column of comments (basically email text)

Answer 1

Update: Assuming df.Description is your column, this might be helpful:

arr_string = df.Description.astype(str).values.tolist()
for val in arr_string:
    for words in val:
            comment = ''.join([comment, words])

Take a look at this .