根据条件合并Dataframe的行

Question

I have a csv file with only one column "notes". I want to merge rows of data-frame based on some condition.

Input_data={'notes':
            ['aaa','bbb','*','hello','**','my name','is xyz',
             '(1)','this is','temp','name',
             '(2)','BTW','how to','solve this',
             '(3)','with python','I don’t want this to be added ',
             'I don’t want this to be added ']}

df_in = pd.DataFrame(Input_data) 

Input looks like this

Output

output_Data={'notes':
             ['aaa','bbb','*hello','**my name is xyz',
              '(1) this is temp name',
              '(2) BTW how to solve this',
              '(3) with python','I don’t want this to be added ',
              'I don’t want this to be added ']}
df_out=pd.DataFrame(output_Data) 

I want to merge the rows with the above row which have either "*" or "(number)" in it. So the output will look like

Other rows which can not be merged should be left. Also, in case of last row as there is no proper way to know up-to what range we can merge lets say just add only one next row I solved this but its very long. Any simpler way

df=pd.DataFrame(Input_data)
notes=[];temp=[];flag='';value='';c=0;chk_star='yes'
for i,row in df.iterrows():
    row[0]=str(row[0])
    if '*' in row[0].strip()[:5] and chk_star=='yes':   
        value=row[0].strip()
        temp=temp+[value]
        value=''
        continue

    if '(' in row[0].strip()[:5]:
        chk_star='no'
        temp=temp+[value]
        value='';c=0
        flag='continue'
        value=row[0].strip()
    if flag=='continue' and '(' not in row[0][:5] : 
        value=value+row[0]
        c=c+1
    if c>4:
        temp=temp+[value] 
        print "111",value,temp
        break
if '' in temp:
    temp.remove('')
df=pd.DataFrame({'notes':temp})     

Answer 1

You can use a mask to avoid the for loop :

df = pd.DataFrame({'row':['aaa','bbb','*','hello','**','my name','is xyz',
         '(1)','this is ','temp ','name',
         '(2)','BTW ','how to ','solve this',
         '(3)','with python ','I don’t want this to be added ',
         'I don’t want this to be added ']})

special = ['*', '**']
for i in range(11):
    special.append('({})'.format(i))

# We find the indexes where we will have to merge
index_to_merge = df[df['row'].isin(special)].index.values
for idx, val in enumerate(index_to_merge):
    if idx != len(index_to_merge)-1:
        df.loc[val, 'row'] += ' ' + df.loc[val+1:index_to_merge[idx+1]-1, 'row'].values.sum()
    else:
        df.loc[index, 'row'] += ' ' + df.loc[index+1:, 'row'].values.sum()

# We delete the rows that we just used to merge
df.drop([x for x in np.array(range(len(df))) if x not in index_to_merge])

Out :

        row
2   * hello
4   ** my nameis xyz
7   (1) this is temp name
11  (2) BTW how to solve this
15  (3) with python I don’t want this to be added ..

---

You could also convert your column into a numpy array and use numpy functions to simplify what you did. First you can use the np.where and np.isin to find the indexes where you will have to merge. That way you don't have to iterate on your whole array using a for loop.

Then you can do the mergures on the corresponding indexes. Finally, you can delete the values that have been merged. Here is what it could look like :

list_to_merge = np.array(['aaa','bbb','*','hello','**','my name','is xyz',
             '(1)','this is','temp','name',
             '(2)','BTW','how to','solve this',
             '(3)','with python','I don’t want this to be added ',
             'I don’t want this to be added '])
special = ['*', '**']
for i in range(11):
    special.append('({})'.format(i))

ix = np.isin(list_to_merge, special)
rows_to_merge = np.where(ix)[0]

# We merge the rows
for index_to_merge in np.where(ix)[0]:
    # Check if there we are not trying to merge with an out of bounds value
    if index_to_merge!=len(list_to_merge)-1:
        list_to_merge[index_to_merge] = list_to_merge[index_to_merge] + ' ' + list_to_merge[index_to_merge+1]

# We delete the rows that have just been used to merge:
rows_to_delete = rows_to_merge +1
list_to_merge = np.delete(list_to_merge, rows_to_delete)

Out :

['aaa', 'bbb', '* hello', '** my name', 'is xyz', '(1) this is',
       'temp', 'name', '(2) BTW', 'how to', 'solve this',
       '(3) with python', 'I don’t want this to be added ',
       'I don’t want this to be added ']

Answer 2

Below solution recognises special characters like *,** and (number) at the start of the of the sentence and starts merging later rows except last row.

import pandas as pd
import re
df = pd.DataFrame({'row':['aaa','bbb','*','hello','**','my name','is xyz',
             '(1)','this is','temp','name',
             '(2)','BTW','how to','solve this',
             '(3)','with python','I don’t want this to be added ',
             'I don’t want this to be added ']})



pattern = "^\(\d+\)|^\*+" #Pattern to identify string starting with (number),*,**.

#print(df)
#Selecting index based on the above pattern
selected_index = df[df["row"].str.contains(re.compile(pattern))].index.values
delete_index = []
for index in selected_index:
    i=1
    #Merging row until next selected index found and add merged rows to delete_index list
    while(index+i not in selected_index and index+i < len(df)-1):
        df.at[index, 'row'] += ' ' + df.at[index+i, 'row']
        delete_index.append(index+i)
        i+=1


df.drop(delete_index,inplace=True)
#print(df)

Output:

    row
0   aaa
1   bbb
2   *hello
4   **my nameis xyz
7   (1)this istempname
11  (2)BTWhow tosolve this
15  (3)with pythonI don’t want this to be added
18  I don’t want this to be added

You can reset index if you want. using df.reset_index()

Answer 3

I think it is easier when you design your logic to separate df_in into 3 parts: top, middle and bottom . Keeping top and bottom intact while joining middle part. Finally, concat 3 parts together into df_out

First, create m1 and m2 masks to separate df_in to 3 parts.

m1 = df_in.notes.str.strip().str.contains(r'^\*+|\(\d+\)$').cummax()
m2 =  ~df_in.notes.str.strip().str.contains(r'^I don’t want this to be added$')
top = df_in[~m1].notes
middle = df_in[m1 & m2].notes
bottom = df_in[~m2].notes

Next, create groupby_mask to group rows and groupby and join :

groupby_mask = middle.str.strip().str.contains(r'^\*+|\(\d+\)$').cumsum()
middle_join = middle.groupby(groupby_mask).agg(' '.join)

Out[3110]:
notes
1                      * hello
2            ** my name is xyz
3        (1) this is temp name
4    (2) BTW how to solve this
5              (3) with python
Name: notes, dtype: object

Finally, use pd.concat to concat top , middle_join , bottom

df_final = pd.concat([top, middle_join, bottom], ignore_index=True).to_frame()

Out[3114]:
                            notes
0                             aaa
1                             bbb
2                         * hello
3               ** my name is xyz
4           (1) this is temp name
5       (2) BTW how to solve this
6                 (3) with python
7  I don’t want this to be added
8  I don’t want this to be added