当 Python 中的值总和为 n 时，如何对列表中的元素求和并使用它们创建子列表

Question

I am kinda stuck.

So here is my scenario.

I have a list of small file(parquet files). My goal is track them and merge them in a more optimal parquet files size.

While i could just read all and run repartition, this wont apply to my use case, since they share location with other already partitioned files(would be to much to handle in terms of data volume).

So i have list That looks like this:

[
[('filepath.parquet',1000),
('filepath.parquet',1000)],

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)],

 [('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)]
]

My Goal is to have max group size parameter that will create sub lists of this lists based on the sum of bytes.

Using my list example with a max_group of 5000 where i have one main list with 3 sub lists i would get:

1 - main list - no change here 1 - sub list 1 would keel all its elements since the sum of the bytes is only 2000 2 - sub list 2 would be split in 2 sub sub lists since the total sum is 8000 and max_group is 5000. eg:

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)],

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)]

• so one sub sub list will be 5000 and the other will be 3000

3 - sub list 3 - will be split in 3 sub sub lists as bellow, again following the same max_group

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)],

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)],

[('filepath.parquet',1000)]

 

So my final list would be:

[ -- main list
    [ -- sub list 
            [ -- sub sub list
            ('filepath.parquet',1000),
            ('filepath.parquet',1000)
            ],

            [
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000)
            ],

            [
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000)
        ]
    ],

    [
            [
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000)
            ],

            [
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000),
            ('filepath.parquet',1000)
            ],
    ],
    [
            [
            ('filepath.parquet',1000)
            ]
    ]
]

So i am trying to do this in python: my code:

lst = [
[('filepath.parquet',1000),
('filepath.parquet',1000)],

[('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)],

 [('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000),
('filepath.parquet',1000)]
]
    
    
max_group  = 5000
i = 0
for k,sublst in enumerate(lst):
    print('entering sublst: ' + str(k))
    for file in sublst:
        f, v = file
        tot = i + v
        print(f)
        while tot <= max_group:
            tot = tot + v

Answer 1

Here you are trying to iterate v which is an integer and get a sum. The following is the sum method.

def sum(*args, **kwargs): # real signature unknown
    """
    Return the sum of a 'start' value (default: 0) plus an iterable of numbers
    
    When the iterable is empty, return the start value.
    This function is intended specifically for use with numeric values and may
    reject non-numeric types.
    """
    pass

You need to pass an array to the sum method.

The following code will give you the intended solution.

max_group = 5000
i = 0
final_list = []
for k, sublst in enumerate(lst):
    print('entering sublst: ' + str(k))
    size = 0
    temp_list = []
    for file in sublst:
        f, v = file
        size += v

        if size > max_group:
            final_list.append(temp_list)
            temp_list = [file]
            size = v
        else:
            temp_list.append(file)

    if len(temp_list) > 0:
        final_list.append(temp_list)

First, it will append the v to the size, but it will not append f to the list. If the size is greater than the max_group, the temp_list will be appended to the final_list and initialize the temp_list with the file. Then it will reinitialize size as well. If the size is less than max_group it will keep appending to the temp_list. At the end of the for loop, it will check the temp_list length and if there are leftover elements in the temp_list, the temp list will be appended to the final_list as well.