How to efficiently select specific columns with condition in python

Question

For example, I have the following 2d array:

ls = [
    [1,2,3,4,'A',5],
    [1,2,3,4,'A',5],
    [1,2,3,4,'A',5],
    [-1,-2,-3,-4,'B',-5],
    [-1,-2,-3,-4,'B',-5],
    [-1,-2,-3,-4,'B',-5]
]

I want to select the 1st, 3rd, 4th column of ls , and respectively save each column into a new list. Moreover, I hope to select conditioned on the 5th column, ie checking whether 'A' or 'B' , as follows:

la1 = [int(x[0]) for x in ls if 'A' in x[4]]
la2 = [int(x[2]) for x in ls if 'A' in x[4]]
la3 = [float(x[3]) for x in ls if 'A' in x[4]]
lb1 = [int(x[0]) for x in ls if 'B' in x[4]]
lb2 = [int(x[2]) for x in ls if 'B' in x[4]]
lb3 = [float(x[3]) for x in ls if 'B' in x[4]]

I know my implementation is not efficient in large arrays. Is there any better implementation? Thank you all for helping me!!!

Answer 1

You can merge your 6 list comprehensions into two:

la1, la2, la3= zip(*((x[0], x[2], float(x[3])) for x in ls if 'A' in x[4]))
lb1, lb2, lb3= zip(*((x[0], x[2], float(x[3])) for x in ls if 'B' in x[4]))

This first creates a list of 3-tuples (x[0], x[2], float(x[3])) , then uses the old zip(*values) trick to transpose it and unpack it into the la1, la2, la3 variables.

---

More efficient than that would be a simple loop:

la1, la2, la3 = [], [], []
lb1, lb2, lb3 = [], [], []
for x in ls:
    if 'A' in x[4]:
        la1.append(x[0])
        la2.append(x[2])
        la3.append(float(x[3]))
    if 'B' in x[4]:
        lb1.append(x[0])
        lb2.append(x[2])
        lb3.append(float(x[3]))

Answer 2

You can try to use numpy , it's highly efficient array library for python:

import numpy as np

ls = np.array([  # wrap ls into numpy array
    [1,2,3,4,'A',5],
    [1,2,3,4,'A',5],
    [1,2,3,4,'A',5],
    [-1,-2,-3,-4,'B',-5],
    [-1,-2,-3,-4,'B',-5],
    [-1,-2,-3,-4,'B',-5]
])

a_rows = ls[:,4] == 'A' # select rows with A in 4-th column
b_rows = ls[:,4] == 'B'
col_1 = ls[:,0]  # select first column
col_3 = ls[:,2]
col_4 = ls[:,3]
la1 = col_1[a_rows]  # first column with respect to the rows with A
la2 = col_3[a_rows]
la3 = col_4[a_rows]
lb1 = col_1[b_rows]
lb2 = col_3[b_rows]
lb3 = col_4[b_rows]

Answer 3

I think storing your lists in a dictionary would be wise if you have many of them, also a for loop might be quicker since you are kind of splitting the data based on a condition:

d = {'la1': [],
     'la3': [],
     'la4': [],
     'lb1': [],
     'lb3': [],
     'lb4': []}

ls = [[1,2,3,4,'A',5],
      [1,2,3,4,'A',5],
      [1,2,3,4,'A',5],
      [-1,-2,-3,-4,'B',-5],
      [-1,-2,-3,-4,'B',-5],
      [-1,-2,-3,-4,'B',-5]]

for sublist in ls:
    if sublist[4] == "A":
        d['la1'].append(int(sublist[0]))
        d['la3'].append(int(sublist[2]))
        d['la4'].append(float(sublist[3]))
    elif sublist[4] == "B":
        d['lb1'].append(int(sublist[0]))
        d['lb3'].append(int(sublist[2]))
        d['lb4'].append(float(sublist[3]))

print (d)

#{'lb4': [-4.0, -4.0, -4.0], 'lb1': [-1, -1, -1], 'la3': [3, 3, 3], 'la4': [4.0, 4.0, 4.0], 'la1': [1, 1, 1], 'lb3': [-3, -3, -3]}

Answer 4

Use numpy arrays They are faster than normal lists Try running each line of the code provided below

ls = np.array([[1,2,3,4,'A',5],[1,2,3,4,'A',5],[1,2,3,4,'A',5],[-1,-2,-3,-4,'B',-5],[-1,-2,-3,-4,'B',-5],[-1,-2,-3,-4,'B',-5]])
filterA = (ls[:,4]=='A')
filterB = (ls[:,4]=='B')
newarrayA=ls[filterA]
newarrayB=ls[filterB]
selectedcolumnsA=newarrayA[:,(0,2,3)]
selectedcolumnsB=newarrayB[:,(0,2,3)]  
la1,la2,la3=selectedcolumnsA[:,0],selectedcolumnsA[:,1],selectedcolumnsA[:,2]
lb1,lb2,lb3=selectedcolumnsB[:,0],selectedcolumnsB[:,1],selectedcolumnsB[:,2]

Hope it helps.If you are uncomfortable with it,Try learning numpy.It will surely help you in future.