Using modin provides different results compared to Pandas default

Question

I am getting different results when I use pandas within modin and when using pandas default

print(selection_weights.head())
  country                      league   Win   DNB  O 1.5  U 4.5
0  Africa       Africa Cup of Nations  3.68  1.86    5.2   1.45
1  Africa   Africa Cup of Nations U17  2.07  1.50    3.3   1.45
2  Africa   Africa Cup of Nations U20  2.07  1.50    3.3   1.45
3  Africa   Africa Cup of Nations U23  2.07  1.50    3.3   1.45
4  Africa  African Championship Women  2.07  1.50    3.3   1.45

print(historical_games.head())
   Unnamed: 0  home_odds  draw_odds  away_odds country            league             datetime        home_team   away_team  home_score  away_score
0           0       1.36       4.31       7.66  Brazil  Copa do Nordeste  2020-02-07 00:00:00     Sport Recife  Imperatriz           2           2
1           1       2.62       3.30       2.48  Brazil  Copa do Nordeste  2020-02-02 22:00:00              ABC  America RN           2           1
2           2       5.19       3.58       1.62  Brazil  Copa do Nordeste  2020-02-02 00:00:00  Frei Paulistano     Nautico           0           2
3           3       2.06       3.16       3.50  Brazil  Copa do Nordeste  2020-02-02 22:00:00      Botafogo PB   Confianca           1           1
4           4       2.19       2.98       3.38  Brazil  Copa do Nordeste  2020-02-02 22:00:00        Fortaleza       Ceara           1           1

When I run the below code in default pandas , the output is as desired:

import pandas as pd

selection_db = historical_games.loc[:, historical_games.columns.intersection(['country', 'league'])]
selection_db = selection_db.drop_duplicates()
selection_db = selection_db.sort_values(['country', 'league'], ascending=[True, True])
selection_db.loc[:, 'Win'] = 1.1
selection_db.loc[:, 'DNB'] = 0.7
selection_db.loc[:, 'O 1.5'] = 3.2
selection_db.loc[:, 'U 4.5'] = 2.2
ids = ['country', 'league']
selection_db = selection_db.set_index(ids)
selection_db.update(selection_weights.drop_duplicates(ids).set_index(ids))
selection_db = selection_db.reset_index()
selection_weights = selection_db
print(selection_weights.head())

  country                      league   Win   DNB  O 1.5  U 4.5
0  Africa       Africa Cup of Nations  3.68  1.86    5.2   1.45
1  Africa   Africa Cup of Nations U17  2.07  1.50    3.3   1.45
2  Africa   Africa Cup of Nations U20  2.07  1.50    3.3   1.45
3  Africa   Africa Cup of Nations U23  2.07  1.50    3.3   1.45
4  Africa  African Championship Women  2.07  1.50    3.3   1.45

But when I run it with modin , I get a different and incorrect output

import os
import ray
ray.init()
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as pd

selection_db = historical_games.loc[:, historical_games.columns.intersection(['country', 'league'])]
selection_db = selection_db.drop_duplicates()
selection_db = selection_db.sort_values(['country', 'league'], ascending=[True, True])
selection_db.loc[:, 'Win'] = 1.1
selection_db.loc[:, 'DNB'] = 0.7
selection_db.loc[:, 'O 1.5'] = 3.2
selection_db.loc[:, 'U 4.5'] = 2.2
ids = ['country', 'league']
selection_db = selection_db.set_index(ids)
selection_db.update(selection_weights.drop_duplicates(ids).set_index(ids))
selection_db = selection_db.reset_index()
selection_weights = selection_db
print(selection_weights.head())

  country  league
0  Africa     2.2
1  Africa     2.2
2  Africa     2.2
3  Africa     2.2
4  Africa     2.2

The problem is that I have to run the function as a part of a big workflow and while I import modin at the start, it performs as expected till this part of the code.

While I am unable to revert to default pandas in between the code or I dont know how to change libraries in between a code.

How do I resolve this situation?

Answer 1

@Harshad, this comment from the Modin GitHub describes how to convert a Modin dataframe to pandas: use df._to_pandas() . Once you have a pandas dataframe, you call any pandas method on it. This other comment from the same issue describes how to convert the pandas dataframe back to a Modin dataframe: call modin.pandas.DataFrame(pandas_dataframe) .

Regarding the Modin errors you are seeing, my guess is that the lines like selection_db.loc[:, 'Win'] = 1.1 where you add columns are raising a KeyError and not changing the Modin dataframe at all. That's a known Modin bug, https://github.com/modin-project/modin/issues/4354 . For example, this works in pandas

import pandas
df = pandas.DataFrame([[1]])
df.loc[:, 'a'] = 3

but if I try the same script with import modin.pandas as pandas and the latest version of Modin (commit c1d5dbd71efb8fb5806fad41959794182780fc25), I get KeyError: array(['a'], dtype='<U1') . Is it possible that you're getting a KeyError and ignoring it?