Increase the speed of my code

Question

i have created the code below, it takes a series of values, and generates 10 numbers between x and r with an average value of 8000

in order to meet the specification to cover the range as well as possible, I also calculated the standard deviation, which is a good measure of spread. So whenever a sample set meets the criteria of mean of 8000, I compared it to previous matches and constantly choose the samples that have the highest std dev (mean always = 8000)

def node_timing(average_block_response_computational_time, min_block_response_computational_time, max_block_response_computational_time):
    sample_count = 10
    num_of_trials = 1

    # print average_block_response_computational_time
    # print min_block_response_computational_time
    # print max_block_response_computational_time

    target_sum = sample_count * average_block_response_computational_time
    samples_list = []
    curr_stdev_max = 0
    for trials in range(num_of_trials):
        samples = [0] * sample_count
        while sum(samples) != target_sum:
            samples = [rd.randint(min_block_response_computational_time, max_block_response_computational_time) for trial in range(sample_count)]
        # print ("Mean: ", st.mean(samples), "Std Dev: ", st.stdev(samples), )
        # print (samples, "\n")
        if st.stdev(samples) > curr_stdev_max:
            curr_stdev_max = st.stdev(samples)
            samples_best = samples[:]
        return samples_best[0]

i take the first value in the list and use this as a timing value, however this code is REALLY slow, i need to call this piece of code several thousand times during the simulation so need to improve the efficency of the code some how

anyone got any suggestions on how to ?

Answer 1

To see where we'd get the best speed improvements, I started by profiling your code.

import cProfile
pr = cProfile.Profile()
pr.enable()
for i in range(100):
    print(node_timing(8000, 7000, 9000))
pr.disable()
pr.print_stats(sort='time')

The top of the results show where your code is spending most of its time:

   23561178 function calls (23561176 primitive calls) in 10.612 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  4502300    3.694    0.000    7.258    0.000 random.py:172(randrange)
  4502300    2.579    0.000    3.563    0.000 random.py:222(_randbelow)
  4502300    1.533    0.000    8.791    0.000 random.py:216(randint)
   450230    1.175    0.000    9.966    0.000 counter.py:19(<listcomp>)
  4608421    0.690    0.000    0.690    0.000 {method 'getrandbits' of '_random.Random' objects}
      100    0.453    0.005   10.596    0.106 counter.py:5(node_timing)
  4502300    0.294    0.000    0.294    0.000 {method 'bit_length' of 'int' objects}
   450930    0.141    0.000    0.150    0.000 {built-in method builtins.sum}
      100    0.016    0.000    0.016    0.000 {built-in method builtins.print}
      600    0.007    0.000    0.025    0.000 statistics.py:105(_sum)
     2200    0.005    0.000    0.006    0.000 fractions.py:84(__new__)
...

From this output, we can see that we're spending ~7.5 seconds (out of 10.6 seconds) generating random numbers. Therefore, the only way to make this noticeably faster is to generate fewer random numbers or generate them faster. You're not using a cryptographic random number generator so I don't have a way to make generating numbers faster. However, we can fudge the algorithm a bit and drastically reduce the number of values we need to generate.

Instead of only accepting samples with a mean of exactly 8000, what if we accepted samples with a mean of 8000 +- 0.1% (then we're taking samples with a mean of 7992 to 8008)? By being a tiny bit inexact, we can drastically speed up the algorithm. I replaced the while condition with:

while abs(sum(samples) - target_sum) > epsilon

Where epsilon = target_sum * 0.001 . Then I ran the script again and got much better profiler numbers.

         232439 function calls (232437 primitive calls) in 0.163 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100    0.032    0.000    0.032    0.000 {built-in method builtins.print}
    31550    0.026    0.000    0.053    0.000 random.py:172(randrange)
    31550    0.019    0.000    0.027    0.000 random.py:222(_randbelow)
    31550    0.011    0.000    0.064    0.000 random.py:216(randint)
     4696    0.010    0.000    0.013    0.000 fractions.py:84(__new__)
     3155    0.008    0.000    0.073    0.000 counter.py:19(<listcomp>)
      600    0.008    0.000    0.039    0.000 statistics.py:105(_sum)
      100    0.006    0.000    0.131    0.001 counter.py:4(node_timing)
    32293    0.005    0.000    0.005    0.000 {method 'getrandbits' of '_random.Random' objects}
     1848    0.004    0.000    0.009    0.000 fractions.py:401(_add)

Allowing the mean to be up to 0.1% off of the target dropped the number of calls to randint by 100x. Naturally, the code also runs 100x faster (and now spends most of its time printing to console).