[英]Speeding up this Python code for large input
I wrote this Python code to do a particular computation in a bigger project and it works fine for smaller values of N
but it doesn't scale up very well for large values and even though I ran it for a number of hours to collect the data, I was wondering if there was a way to speed this up 我编写了此Python代码,以便在较大的项目中进行特定的计算,并且对于较小的N
值也可以正常工作,但是对于较大的值,即使我运行了几个小时来收集数据,它也无法很好地扩展,我想知道是否有一种方法可以加快速度
import numpy as np
def FillArray(arr):
while(0 in arr):
ind1 = np.random.randint(0,N)
if(arr[ind1]==0):
if(ind1==0):
arr[ind1] = 1
arr[ind1+1] = 2
elif(ind1==len(arr)-1):
arr[ind1] = 1
arr[ind1-1] = 2
else:
arr[ind1] = 1
arr[ind1+1] = 2
arr[ind1-1] = 2
else:
continue
return arr
N=50000
dist = []
for i in range(1000):
arr = [0 for x in range(N)]
dist.append(Fillarr(arr).count(2))
For N = 50,000
, it currently takes slightly over a minute on my computer for one iteration to fill the array. 对于N = 50,000
,当前在我的计算机上需要一分钟多的时间才能进行一次迭代以填充阵列。 So if I want to simulate this, lets say, a 1000 times, it takes many hours. 因此,如果我要模拟这个,可以说是1000次,那要花很多小时。 Is there something I can do to speed this up? 我可以做些什么来加快速度吗?
Edit 1: I forgot to mention what it actually does. 编辑1:我忘记提及它的实际作用。 I have a list of length N
and I initialize it by having zeros in each entry. 我有一个长度为N
的列表,并通过在每个条目中添加零来对其进行初始化。 Then I pick a random number between 0
and N
and if that index of the list has a zero, I replace it by 1
and its neighboring indices by 2
to indicate they are not filled by 1
but they can't be filled again. 然后,我选择一个介于0
和N
之间的随机数,如果列表的索引为零,则将其替换为1
并将其相邻的索引替换为2
以指示它们未被1
填充,但无法再次填充。 I keep doing this till I populate the whole list by 1
and 2
and then I count how many of the entries contain 2
which is the result of this computation. 我一直这样做,直到用1
和2
填充整个列表,然后计算多少个条目包含2
,这是此计算的结果。 Thus I want to find out if I fill an array randomly with this constraint, how many entries will not be filled. 因此,我想找出是否使用此约束随机填充数组,将不填充多少个条目。
Obviously I do not claim that this is the most efficient way find this number so I am hoping that perhaps there is a better alternative way if this code can't be speeded up. 显然,我并不认为这是找到此数字的最有效方法,因此我希望,如果无法加快此代码的速度,也许会有更好的选择。
As @SylvainLeroux noted in the comments, the approach of trying to find what zero you're going to change by drawing a random location and hoping it's zero is going to slow down when you start running out of zeros. 正如@SylvainLeroux在评论中指出的那样,当您开始用尽零位时,试图通过绘制随机位置来找到要更改的零并希望其为零的方法将会变慢。 Simply choosing from the ones you know are going to be zero will speed it up dramatically. 只需从您知道将为零的值中进行选择,即可大大加快速度。 Something like 就像是
def faster(N):
# pad on each side
arr = np.zeros(N+2)
arr[0] = arr[-1] = -1 # ignore edges
while True:
# zeros left
zero_locations = np.where(arr == 0)[0]
if not len(zero_locations):
break # we're done
np.random.shuffle(zero_locations)
for zloc in zero_locations:
if arr[zloc] == 0:
arr[zloc-1:zloc+2] = [2, 1, 2]
return arr[1:-1] # remove edges
will be much faster (times on my old notebook): 将更快(在我的旧笔记本上的时间):
>>> %timeit faster(50000)
10 loops, best of 3: 105 ms per loop
>>> %time [(faster(50000) == 2).sum() for i in range(1000)]
CPU times: user 1min 46s, sys: 4 ms, total: 1min 46s
Wall time: 1min 46s
We could improve this by vectorizing more of the computation, but depending on your constraints this might already suffice. 我们可以通过对更多计算进行矢量化来改善此情况,但是根据您的约束,这可能已经足够。
First I will reformulate the problem from tri-variate to bi-variate. 首先,我将把问题从三变量重新表述为双变量。 What you are doing is spliting the vector of length N into two smaller vectors at random point k. 您正在做的是在随机点k处将长度为N的向量分成两个较小的向量。
Lets assume that you start with a vector of zeros, then you put '1' at randomly selected k and from there take two smaller vectors of zeros - [0..k-2] & [k+2.. N-1]. 假设您从零的向量开始,然后在随机选择的k处放置“ 1”,然后从中选取两个较小的零向量-[0..k-2]和[k + 2 .. N-1] 。 No need for 3rd state. 不需要第三状态。 You repeat the process until exhaustion - when you are left with vectors containing only one element. 重复此过程,直到用尽为止-剩下的向量仅包含一个元素。
Using recusion this is reasonably fast even on my iPad mini with Pythonista. 使用recusion,即使在带Pythonista的iPad mini上,这也相当快。
import numpy as np
from random import randint
def SplitArray(l, r):
while(l < r):
k = randint(l, r)
arr[k] = 1
return SplitArray(l, k-2) + [k] + SplitArray(k+2, r)
return []
N = 50000
L = 1000
dist=np.zeros(L)
for i in xrange(L):
arr = [0 for x in xrange(N)]
SplitArray(0, N-1)
dist[i] = arr.count(0)
print dist, np.mean(dist), np.std(dist)
However if you would like to make it really fast then bivariate problem could be coded very effectively and naturally as bit arrays instead of storing 1 and 0 in arrays of integers or worse floats in numpy arrays. 但是,如果您想使其变快,则可以非常有效且自然地将双变量问题编码为位数组,而不是将1和0存储在整数数组中,或更糟糕的是将其存储在numpy数组中。 The bit manipulation should be quick and in some you easily could get close to machine level speed. 钻头操作应该很快,在某些情况下,您很容易接近机器水平的速度。
Something along the line: (this is an idea not optimal code) 大致思路:(这是一个主意,而不是最佳代码)
from bitarray import BitArray
from random import randint
import numpy as np
def SplitArray(l, r):
while(l < r):
k = randint(l, r)
arr.set_bit(k)
return SplitArray(l, k-2) + [k] + SplitArray(k+2, r)
return []
def count0(ba):
i = 0
for n in xrange(1, N):
if ba.get_bit(n) == 0:
i += 1
return i
N = 50000
L = 1000
dist = np.zeros(L)
for i in xrange(L):
arr = BitArray(N, initialize = 0)
SplitArray(1, N)
dist[i] = count0(arr)
print np.mean(dist), np.std(dist)
The solution converges very nicely so perhaps half an hour spent looking for an analytical solution would make this whole MC excercise unnecessary? 该解决方案收敛非常好,所以也许花半小时寻找一种分析解决方案将使整个MC职位变得不必要?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.