Multi-threading for embedded loops in python

Question

I have a python code which does some manipulation of data contained in a Shapefile.

Among others stuffs, the code does this:

 xxx=0
 for i in channels:
      ptsi=mat(shapes[i].points)
      xx = ptsi[:,0]
      yy = ptsi[:,1]

      nanx=argwhere(isnan(xx))      
      nany=argwhere(isnan(yy))

      if (nanx == nany and len(nanx) != 0):
          xx[nanx] = []
          yy[nany] = []
      elif (len(nanx) != 0 or len(nany) != 0):
          xx = []
          yy = []

      if (len(xx) > 0):          
       yyy = 0 
       dd = str(i*100/N_channels) + '%\r\r'
      # os.write(1,dd)
       dist = zeros(len(xx)-1)

       dist = dist_min
       for j in channels:
           pts=mat(shapes[j].points)
           xx2 = pts[:,0]
           yy2 = pts[:,1]

           nanx=argwhere(isnan(xx2))          
           nany=argwhere(isnan(yy2))

           if (nanx == nany and len(nanx) != 0):
            xx2[nanx] = []
            yy2[nany] = []
           elif (len(nanx) != 0 or len(nany) != 0):
            xx2 = []
            yy2 = []
           if (len(xx2) > 0):  
            if (i != j): 
               ds = gaa(xx,yy,xx2[0],yy2[0])
               de = gaa(xx,yy,xx2[-1],yy2[-1])
               nande = ~isnan(de)
               nands = ~isnan(ds)
               fe = np.argwhere(de<=dist)
               fs = np.argwhere(ds<=dist)
               nozeroe = array(where(de != 0))
               nozeroe = nozeroe[0]
               nozeros = array(where(ds != 0))
               nozeros = nozeros[0] 
               if(fs.size >0):
                    iis=array(np.argwhere(array(ds==min(ds))))

                    iis = iis.flatten()
                    iis = iis[0]
                    p1 = xxx + iis 
                    p2 = yyy
                    G.add_edge(p2,p1,weight=w_con)

               elif (fe.size > 0):
                    iie=array(np.argwhere(array(de==min(de))))
                    iie = iie.flatten()
                    iie = iie[0]
                    p1 = xxx + iie
                    p2 = yyy  + len(pts) -1
                    G.add_edge(p2,p1,weight=w_con)

           yyy = yyy + len(pts)

      xxx = xxx + len(ptsi)

Basically, the code scans over polylines and search for a minimal distance in order to merge then in a common graph (using Networkx). This part works fine except that this is really slow as I deal with more than 100's of thousand of objects (It takes roughly 20 hours in its current version).

These embedded loops are not efficient, so I'd like to know, if using multithreading could be helpful and if so, how can I modify this very part of the code? I'm fine with CUDA or OpenCL if it can help.

Thanks for any feedback!

Answer 1

由于使用全局解释器锁 ，Python代码无法通过多线程完全利用多个内核，并且需要使用多重处理才能充分利用多个内核。

Answer 2

More threads will not help; more processes might, but if you can use CUDA, that would probably be a good move. A thread is just a way to allow multiple things to share a processes CPU time, it does not speed up slow code, but will just slow you down more.