python - 巨大的 for 循环很慢，有没有更快的方法来处理它

Question

I'm running a lots of data we speak about 10-15 mil. multiply times in for loop, and i can found out of its my for loop there are slow eg.

i disover the problem when i test 100 rows, then 1000 and then 25.000 rows every time i encrse my numbers of rows its taken much longer time to run.

i'm using numpy today to calculate a lots of prices and its working perfect, so now i hit the wall with so much data there shut running multi time to restructuer the data befure i'm ready to return it to a final array/dict.

a orther issue its when i run a for in a for loop, its taken much more time then a for ofc i need to run "for 1 rows * for 2 rows"

my case is.

1. 1.200.000 products
   1. price rules there can be hit (25-30 diff price rules)
   2. select the right price rule and do stuff based on htis rule
   3. restruter the data using for loop before using numpy
   4. calculate a price to the main product price by using numpy
2. run the product pricerule group agenst 10-15 price groups for each product
   1. restructer the data and prepare in a for loop before using numpy
   2. append the price to main product price array in for loop
3. restruter all data and prepare to return it in my array/dict so i can use it later (single / multi product calc)

hope you understand what i want, and why this will take so much time, and hope there are one there can help me to find a faster way to calculate so much data in.

i have thinking on multitread option, but i think i need to fix the main-for-loop issue before i go to the next performance setup

a very basic sample of my case how to hit this loop in a loop hell

import datetime
start_time = datetime.datetime.now()

product = []
group = []
final_collect = []

for test_product in range(25000):
    product.append({'title': test_product})

for (inx, item) in enumerate(product):
    group.append({
        'product' : item,
        'group-data' : []
    })

    for test_group in range(10):
        group[inx]['group-data'].append({'group' : test_group, 'price' : 100.0})
        print(inx, test_group)

as you can see this one will take around 2-3 sec to run 250.000 loops, when we speak 25.000 products, if we go to run it on 1.200.000 mil. * 10 groups its 12.000.000 loops each and i do it multiply time so its will take long time. but there shut be a faster way around this issue?

Answer 1

run2 below generates a 30% improvement versus equivalent run1 (which you provided). Output is identical.

While this may not be "plug and play" for your use case, it demonstrates some of the tricks you can use to improve performance.

import datetime

def run1(n):
    start_time = datetime.datetime.now()

    product = []
    group = []
    final_collect = []

    for test_product in range(n):
        product.append({'title': test_product})

    for (inx, item) in enumerate(product):
        group.append({'product': item,
                      'group-data': []})

        for test_group in range(10):
            group[inx]['group-data'].append({'group': test_group, 'price': 100.0})

    return group

def run2(n):
    start_time = datetime.datetime.now()

    group = [{'product': {'title': i},
              'group-data': [{'group': test_group, 'price': 100.0} for test_group in range(10)]} \
              for i in range(n)]

    return group

assert run1(10) == run2(10)

%timeit run1(50000)  # 1 loop, best of 3: 372 ms per loop
%timeit run2(50000)  # 1 loop, best of 3: 260 ms per loop