How to add a row at the top of each category by concatenating the new data?

Question

I have 2 datasets (in CSV format) with different size such as follow:

df_old:

index category  text
 0    spam      you win much money
 1    spam      you win 7000 car
 2    spam      you are the winner of the game
 3    not_spam  the weather in Chicago is nice
 4    not_spam  Paris is the capital of France
 5    not_spam  pizza is an Italian food
 6    neutral   we have a party now
 7    neutral   they are driving to downtown
 

df_new:

index category  text
 12   spam      we are the cheapest insurance
 14   spam      London is the capital of Canada
 15   not_spam  no more raining in winter
 25   not_spam  the soccer game plays on HBO
 31   neutral   construction will be done
 

I would like to concatenate df_new to the df_old but in the way that df_new goes on top of df_old 's each category. The expected output should be:

df_concat:

index category  text
 12   spam      we are the cheapest insurance
 14   spam      London is the capital of Canada
 0    spam      you win much money
 1    spam      you win 7000 car
 2    spam      you are the winner of the game
 15   not_spam  no more raining in winter
 25   not_spam  the soccer game plays on HBO
 3    not_spam  the weather in Chicago is nice
 4    not_spam  Paris is the capital of France
 5    not_spam  pizza is an Italian food
 31   neutral   construction will be done
 6    neutral   we have a party now
 7    neutral   they are driving to downtown

I tried this , and it not what i want as it is adding everything to the top, not on each category.

Answer 1

You can try:

(pd.concat([df_new,df_old], sort=False)
   .sort_values('category', ascending=False, kind='mergesort')
)

Output:

   index  category                             text
0     12      spam    we are the cheapest insurance
1     14      spam  London is the capital of Canada
0      0      spam               you win much money
1      1      spam                 you win 7000 car
2      2      spam   you are the winner of the game
2     15  not_spam        no more raining in winter
3     25  not_spam     the soccer game plays on HBO
3      3  not_spam   the weather in Chicago is nice
4      4  not_spam   Paris is the capital of France
5      5  not_spam         pizza is an Italian food
4     31   neutral        construction will be done
6      6   neutral              we have a party now
7      7   neutral     they are driving to downtown