根据其他列表中的值更新列表

Question

I have a list of lists, each list contains four elements, and the elements represent id , age , val1 , val2 . I am manipulating each list in such a way that the val1 and val2 values of that list always depend on the most recent values seen in the previous lists. The previous lists for a list are those lists for which the age difference is not less than timeDelta . The list of lists are in sorted order by age.

My code is working perfect but it is slow. I feel that the line marked * * is generating too many lists of lists and can be avoided, by keep on deleting the lists from the begining one I know that the age difference of a list with the next list is more than timeDelta .

myList = [
          [1,   20, '',     'x'],
          [1,   25, 's',    ''],
          [1,   26, '',     'e'],
          [1,   30, 'd',    's'],
          [1,   50, 'd',    'd'],
          [1,   52, 'f',    'g']
          ]


age_Idx =1
timeDelta = 10

for i in range(len(myList))[1:]:
    newList = myList[:i+1] #Subset of lists.  #********
    respList = newList.pop(-1) 
    currage = float(respList[age_Idx])
    retval = collapseListTogether(newList, age_Idx, currage, timeDelta)
    if(len(retval) == 0):
        continue
    retval[0:2] = respList[0:2]
    print(retval)

def collapseListTogether(li, age_Idx, currage, timeDelta):
    finalList = []
    for xl in reversed(li) :
        #print(xl)
        oldage = float(xl[age_Idx])
        if ((currage-timeDelta) <= oldage < currage):
            finalList.append(xl)
        else:
            break
    return([reduce(lambda a, b: b or a, tup) for tup in zip(*finalList[::-1])])

Example

[1, 20, '',     'x'] ==> Not dependent on anything. Skip this list
[1, 25, 's',    '']    == > [1, 25, '', 'x'] 
[1, 26, '',     'e']   ==>  [1, 26, 's', 'x']
[1, 30, 'd',    's']   ==>  [1, 30, 's', 'e']
[1, 50, 'd',    'd']   ==>  Age difference (50-30 = 20) which is more than 10 
[1, 52, 'f',    'g']   ==>  [1, 52, 'd', 'd']

Answer 1

I'm just rewriting your data structure and your code:

from collections import namedtuple
Record = namedtuple('Record', ['id', 'age', 'val1', 'val2'])
myList = [
      Record._make([1,   20, '',     'x']),
      Record._make([1,   25, 's',    '']),
      Record._make([1,   26, '',     'e']),
      Record._make([1,   30, 'd',    's']),
      Record._make([1,   50, 'd',    'd']),
      Record._make([1,   52, 'f',    'g'])
]

timeDelta = 10

for i in range(1, len(myList)):
    subList = list(myList[:i+1])
    rec = supList.pop(-1) 
    age = float(rec.age)
    retval = collapseListTogether(subList, age, timeDelta)
    if len(retval) == 0:
        continue
    retval.id, retval.age = rec.id, rec.age
    print(retval)

def collapseListTogether(lst, age, tdelta):
    finalLst = []
    [finalLst.append(ele) if age - float(ele.age) <= tdelta and age > float(ele.age)
     else None for ele in lst]
    return([reduce(lambda a, b: b or a, tup) for tup in zip(*finalLst[::-1])])

Your code is not readable to me. I did not change the logic, but just modify places for performance.

One of the way out is to replace your 4-element list with tuple, even better with namedtuple, which is a famous high-performance container in Python. Also, for-loop should be avoided in interpreted languages. In python, one would use comprehensions instead of for-loop if possible to enhance performance. Your list is not too large, so time earned in efficient line interpreting should be more than that in breaking.

To me, your code should not work, but I am not sure.

Answer 2

Assuming your example is correct, I see no reason you can't do this in a single pass, since they're sorted by age. If the last sublist you inspected has too great a difference, you know nothing earlier will count, so you should just leave the current sublist unmodified.

previous_age = None
previous_val1 = ''
previous_val2 = ''

for sublist in myList:
    age = sublist[1]
    latest_val1 = sublist[2]
    latest_val2 = sublist[3]
    if previous_age is not None and ((age - previous_age) <= timeDelta):
        # there is at least one previous list            
        sublist[2] = previous_val1
        sublist[3] = previous_val2
    previous_age = age
    previous_val1 = latest_val1 or previous_val1
    previous_val2 = latest_val2 or previous_val2

When testing, that code produces this modified value for your initial myList:

[[1, 20, '', 'x'],
 [1, 25, '', 'x'],
 [1, 26, 's', 'x'],
 [1, 30, 's', 'e'],
 [1, 50, 'd', 'd'],
 [1, 52, 'd', 'd']]

It's a straightforward modification to build a new list rather than edit one in place, or to entirely omit the skipped lines rather than just leave them unchanged.

reduce and list comprehensions are powerful tools, but they're not right for all problems.