StopIteration issue in pandas dataframe with dictionary python

Question

I have 3 column (DM1_ID, DM2_ID, pairs) pandas dataframe with 1 million records.Also, I have a dictionary containing key and multiple values. The function check the dictionary values and get the key and put that key in new_ID field. Function working fine for small part of pandas dataframe but when I applied it to whole dataframe it'll giving me "StopIteration" error.

        DM1_ID      DM2_ID       pairs
86503   11945.0     11945.0     [11945.0, 11945.0]
86504   11945.0     362380.0    [11945.0, 362380.0]
86505   11945.0     538395.0    [11945.0, 538395.0]
86506   538395.0    591587.0    [11945.0, 591587.0]
86507   11946.0     11946.0     [11946.0, 11946.0]
86508   362380.0    200589       [362380.0,  200589.0]
86509   564785.0    11946.0     [564785.0, 11946.0]

f = lambda x: next(k for k,v in jdic.items() if any(i in v for i in x))

jdic = {10045: [1, 6, 7,10045, 15, 45, 55, 80], 11945: [11945, 362380,20589, 10, 27, 538395, 591587], 3: [3, 21, 28, 32, 35], 11946: [11946, 39, 564785]}


largeFile13000['new_ID'] = largeFile13000['pairs'].apply(f)
largeFile13000.drop('pairs', axis=1, inplace=True)
largeFile13000.head()


# final result I'm expecting is
        DM1_ID     DM2_ID        new_ID
86503   11945.0    11945.0       11945
86504   11945.0    362380.0      11945
86505   11945.0    538395.0      11945
86506   538395.0   591587.0      11945
86507   11946.0    11946.0       11946  
86508   362380.0   200589        11945
86509   564785.0   11946.0       11946




# error

StopIteration                             Traceback (most recent call last)
<ipython-input-14-ddbcd19d6baa> in <module>
----> 1 largeFile13000['new_ID'] = largeFile13000['pairs'].apply(f)
      2 largeFile13000.drop('pairs', axis=1, inplace=True)
      3 largeFile13000.head()

c:\users\ravindu\appdata\local\programs\python\python37\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds)
   3589             else:
   3590                 values = self.astype(object).values
-> 3591                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   3592 
   3593         if len(mapped) and isinstance(mapped[0], Series):

pandas\_libs\lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-12-b4ce01c34c30> in <lambda>(x)
----> 1 f = lambda x: next(k for k,v in jdic.items() if any(i in v for i in x))

StopIteration: 

Does any one could help me to resolve this issue? Thanks in advance.

Answer 1

From your data, essentially you just need to look up one column, say "DM1_ID", as the corresponding "DM2_ID" should belong to the same key in jdic. In this case, it's quite easy to do. I would just reverse your dictionary.

jdic = {10045: [1, 6, 7,10045, 15, 45, 55, 80], 11945: [11945, 362380,20589, 10, 27, 538395, 591587], 3: [3, 21, 28, 32, 35], 11946: [11946, 39, 564785]}

ndic = {}
for key in jdic:
    for i in jdic[key]:
        ndic[i] = key

And then apply the condition.

largeFile13000['new_ID'] = largeFile13000['DM1_ID'].apply(lambda x: ndic[x])

By the way, I don't know whether you have any specific reasons to construct the dictionary jdic in that way. For this apparently many-to-one relationship, it's much better to use the 'many' side as the key.