How to concatenate / join these three dataframes

Question

I have three dataframes df_Male , df_female , Df_TransGender

sample dataframe

df_Male

continent   avg_count_country   avg_age
  Asia          55                5
  Africa        65                10
  Europe        75                8

df_Female

continent   avg_count_country   avg_age
  Asia          50                7
  Africa        60                12
  Europe        70                0

df_Transgender

continent   avg_count_country   avg_age
  Asia          30                6
  Africa        40                11
  America       80                10

Now I am concatenating like this below

frames = [df_Male, df_Female, df_Transgender]
df = pd.concat(frames, keys=['Male', 'Female', 'Transgender'])

As you can see America is present in df_transgender , same wise Europe is present in df_Male and df_Female

So I have to concat it in a way so that it looks like below but not manual as there can be huge number of rows

              continent  avg_count_country  avg_age
Male        0      Asia                 55        5
            1    Africa                 65       10
            2    Europe                 75        8
            3    America                 0        0
Female      0      Asia                 50        7
            1    Africa                 60       12
            2    Europe                 70        0
            3    America                 0        0
Transgender 0      Asia                 30        6
            1    Africa                 40       11
            2    America                80       10
            3    Europe                 0         0

So for other continent values avg_count_country and avg_age should be 0

Answer 1

You can add a "Gender" column before concatenating.

We use Categorical Data with groupby to calculate the Cartesian product. This should also yield performance benefits.

df = pd.concat([df_Male.assign(gender='Male'),
                df_Female.assign(gender='Female'),
                df_Transgender.assign(gender='Transgender')])

for col in ['gender', 'continent']:
    df[col] = df[col].astype('category')

res = df.groupby(['gender', 'continent']).first().fillna(0).astype(int)

print(res)

                       avg_count_country  avg_age
gender      continent                            
Female      Africa                    60       12
            America                    0        0
            Asia                      50        7
            Europe                    70        0
Male        Africa                    65       10
            America                    0        0
            Asia                      55        5
            Europe                    75        8
Transgender Africa                    40       11
            America                   80       10
            Asia                      30        6
            Europe                     0        0

Answer 2

You can reindex a bit.

from itertools import product

# Get rid of that number in the index, not sure why you'd need it
df.index = df.index.droplevel(-1)
# Add continents to the index
df = df.set_index('continent', append=True)

# Determine product of indices
ids = list(product(df.index.get_level_values(0).unique(), df.index.get_level_values(1).unique()))

# Reindex and fill missing with 0
df = df.reindex(ids).fillna(0).reset_index(level=-1)

df is now:

            continent  avg_count_country  avg_age
Male             Asia               55.0      5.0
Male           Africa               65.0     10.0
Male           Europe               75.0      8.0
Male          America                0.0      0.0
Female           Asia               50.0      7.0
Female         Africa               60.0     12.0
Female         Europe               70.0      0.0
Female        America                0.0      0.0
Transgender      Asia               30.0      6.0
Transgender    Africa               40.0     11.0
Transgender    Europe                0.0      0.0
Transgender   America               80.0     10.0

If you want that other numeric index, then you can just do: df.groupby(df.index).cumcount() to number the values in each group.

Answer 3

Making use of DataFrame.pivot , a slight modification to @jpp's answer allows you to avoid having to manually manipulate indices:

df = pd.concat([df_Male.assign(gender='Male'),
                df_Female.assign(gender='Female'),
                df_Transgender.assign(gender='Transgender')])

df.pivot('gender', 'continent').fillna(0).stack().astype(int)

                       avg_count_country  avg_age
gender      continent
Female      Africa                    60       12
            America                    0        0
            Asia                      50        7
            Europe                    70        0
Male        Africa                    65       10
            America                    0        0
            Asia                      55        5
            Europe                    75        8
Transgender Africa                    40       11
            America                   80       10
            Asia                      30        6
            Europe                     0        0