无法比较两个数据帧

Question

I have two Dataframes, df1 and df2 with same structure. I want to find common rows between them using df1.merge(df2) but there is one row I am facing issue with:

>>> df2
  reference_period analyzed_domain account is_misc total_estimated_visits  total_estimated_monthly_unique_visitors  total_estimated_visit_duration  total_estimated_pageviews  estimated_deduplicated_audience
0       2017-11-01          abc     xyz       0                   1000                                    278.0                          5788.0                    80159.0                              0.0
>>> df1=df1.head(1)
>>> df1
  reference_period analyzed_domain account is_misc total_estimated_visits  total_estimated_monthly_unique_visitors  total_estimated_visit_duration  total_estimated_pageviews  estimated_deduplicated_audience
0       2017-11-01          abc     xyz       0                   1000                                    278.0                          5788.0                    80159.0                              0.0
>>> df1==df2
   reference_period  analyzed_domain  account  is_misc  total_estimated_visits  total_estimated_monthly_unique_visitors  total_estimated_visit_duration  total_estimated_pageviews  estimated_deduplicated_audience
0              True             True     True    False                    True                                     True                            True                       True                             True
>>> df1.dtypes
reference_period                           datetime64[ns]
analyzed_domain                                    object
account                                            object
is_misc                                            object
total_estimated_visits                             object
total_estimated_monthly_unique_visitors           float64
total_estimated_visit_duration                    float64
total_estimated_pageviews                         float64
estimated_deduplicated_audience                   float64
dtype: object
>>> df2.dtypes
reference_period                           datetime64[ns]
analyzed_domain                                    object
account                                            object
is_misc                                            object
total_estimated_visits                             object
total_estimated_monthly_unique_visitors           float64
total_estimated_visit_duration                    float64
total_estimated_pageviews                         float64
estimated_deduplicated_audience                   float64
dtype: object

I am not sure why python is not able to equate is_misc column. Could someone please help? Thanks

Answer 1

Pandas dtype object is either str or mixed. So it can be either text or mixed numeric and non-numeric values. In either df1 or df2 , the 0 value for the is_misc column is most likely string type, so you can convert both of them to either string or int, and then run the comparison again, which will then equal True . try this:

df1['is_misc'] = df1['is_misc'].astype(str).astype(int)
df2['is_misc'] = df2['is_misc'].astype(str).astype(int)

And then compare again:

print(df1 == df2)

Answer 2

Gustav Rasmussen ans will work

i got the same problem but i had a string with decimals (eg '5.0') in first dataframe and integer in 2nd dataframe ( eg 5)

i solved the follwing way

df1['column'] = df1['column'].astype(float).astype(int)

df2['column'] = df2['column'].astype(float).astype(int)

and compare

df1==df2