将系列作为新列添加到 pandas dataframe 时缺少行

Question

I have two series, s1 and s2, defined through:

s1 = pd.Series({1: 10, 2: 20, 3: 30, 4: 40}, name='s1')
s2 = pd.Series({3: 35, 4: 45, 5: 55, 6: 65}, name='s2')

They look like this:

1    10
2    20
3    30
4    40
Name: s1, dtype: int64

3    35
4    45
5    55
6    65
Name: s2, dtype: int64

I am trying to create a dataframe that will have s1 and s2 as two columns, with the index a combination of the two series indexes. But a simple assignment doesn't work:

df = pd.DataFrame()
df['s1'] = s1
df['s2'] = s2

The resulting dataframe has the index from s1 but misses the rows from s2 for which the index is not in s1:

   s1    s2
1  10   NaN
2  20   NaN
3  30  35.0
4  40  45.0

Why is that? It seems somewhat counter-intuitive.

Note - a proposed solution is:

df = pd.concat((s1, s2), axis=1)

which gives the expected result:

     s1    s2
1  10.0   NaN
2  20.0   NaN
3  30.0  35.0
4  40.0  45.0
5   NaN  55.0
6   NaN  65.0

But I am nevertheless curious why a simple column assignment doesn't work.

Answer 1

Its because its matching on index, and s2 starts at 3

df = pd.DataFrame()
df['s1'] = s1
df['s2'] = s2

This is setting the shape based on s1, then matching s2 data to the s1 df shape.

df with s1
   s1
1  10
2  20
3  30
4  40

df with s2
   s2
3  35
4  45
5  55
6  65

Answer 2

This is because after you assign a Series to a column of an empty dataframe, the index of dataframe is created align with the Series.

Then you assign another Series to a new column of the one-column dataframe, only indexes in dataframe will be considered.

If you try to assign s2 before s1

   s2    s1
3  35  30.0
4  45  40.0
5  55   NaN
6  65   NaN

print(df)

   s2    s1
3  35  30.0
4  45  40.0
5  55   NaN
6  65   NaN

Answer 3

Since the 2 series have different indices, assigning either series before the other (whether s1 before s2 or s2 before s1) to the empty dataframe would still cause you missing rows. This is because the dataframe index of an empty dataframe is automatically set to that of the first series assigned to it. As a result, when the second series is assigned to the dataframe, it will just take the rows aligning with its current index (just recently set to the index of s1) and ignore rows from the remaining portion of s2 index not common with s1.

There is one remedy to make the 2 statements assigning s1 and s2 to df working as you expect:

df = pd.DataFrame(index=s1.index.union(s2.index))

By presetting the dataframe index to be the union of s1.index and s2.index s1.index.union(s2.index) , you will get your desired result:

df['s1'] = s1
df['s2'] = s2


print(df)

     s1    s2
1  10.0   NaN
2  20.0   NaN
3  30.0  35.0
4  40.0  45.0
5   NaN  55.0
6   NaN  65.0

Breaking down the intermediate steps, you will see interesting result:

df = pd.DataFrame(index=s1.index.union(s2.index))
df['s1'] = s1


print(df)

     s1
1  10.0
2  20.0
3  30.0
4  40.0
5   NaN
6   NaN   

Here, before assigning s2, you can still see index 5 6 (which is part of s2 only) after assigning only s1 and before assigning s2. The corresponding values for index 5 6 are NaN . This is because we have already defined the empty dataframe df with index being union of s1 and s2 while s2 has still not yet assigned to it.

If you want to only modify the dataframe index on the fly after s1 has been assigned to the empty dataframe which has not set with the index= parameter, you can do it by df.reindex() , as follows:

df = pd.DataFrame()                            # without the index= parameter
df['s1'] = s1
df = df.reindex(s1.index.union(s2.index))      # Use reindex()



print(df)

     s1    s2
1  10.0   NaN
2  20.0   NaN
3  30.0  35.0
4  40.0  45.0
5   NaN  55.0
6   NaN  65.0