如何根据多个索引和多个条件过滤一行？

Question

I have a file which looks like this:

#This is TEST-data
2020-09-07T00:00:03.230+02:00,ID-10,3,London,Manchester,London,1,1,1
2020-09-07T00:00:03.230+02:00,ID-10,3,London,London,Manchester,1,1
2020-09-07T00:00:03.230+02:00,ID-20,2,London,London,1,1
2020-09-07T00:00:03.230+02:00,ID-20,2,London,London1,1
2020-09-07T00:00:03.230+02:00,ID-30,3,Madrid,Sevila,Sevilla,1,1,1
2020-09-07T00:00:03.230+02:00,ID-30,GGG,Madrid,Sevilla,Madrid,1
2020-09-07T00:00:03.230+02:00,ID-40,GGG,Madrid,Barcelona,1,1,1,1
2020-09-07T00:00:03.230+02:00
2020-09-07T00:00:03.230+02:00

Index[2] in each row shows how much cities are present in that specific row. So the first row has value 3 for index[2] , which are London, Manchester, London.

I am trying to do the following:

For every row I need to check if any of row [3] + the cities mentioned after it (based on the number of cities) are present in cities_to_filter . But this only needs to be done if row[2] is a number. I also need to tackle the fact that some rows contain less then 2 items.

This is my code:

path = r'c:\data\ELK\Desktop\test_data_countries.txt'

cities_to_filter = ['Sevilla', 'Manchester']

def filter_row(row):
    if row[2].isdigit():
        amount_of_cities = int(row[2]) if len(row) > 2 else True
        
    cities_to_check = row[3:3+amount_of_cities]
    
    condition_1 =  any(city in cities_to_check for city in cities_to_filter)    
    return condition_1

with open (path, 'r') as output_file:
    reader = csv.reader(output_file, delimiter = ',')
    next(reader)
    for row in reader:
        if filter_row(row):
            print(row)

Right now I receive the following error:

UnboundLocalError: local variable 'condition_1' `referenced before assignment`

Answer 1

You could do something like this:

import sys

def filter_row(row):
    '''Returns True if the row should be removed'''
    if len(row) > 2:
        if row[2].isdigit():
            amount_of_cities = int(row[2]) 
            cities_to_check = row[3:3+amount_of_cities]
        else:
            # don't have valid city count, just try the rest of the row
            cities_to_check = row[3:]
        return any(city in cities_to_check for city in cities_to_filter)

    print(f'Invalid row: {row}', file=sys.stderr))
    return True

with open (path, 'r') as input_file:
    reader = csv.reader(input_file, delimiter = ',')
    next(reader)
    for row in reader:
        if filter_row(row):
            print(row)

In filter() the row length is checked to ensure that a possible city count in row[2] is present. If the count is a number it is used to calculate the upper bound to extract the cities to check. Otherwise the row from index 3 to the end of the row is processed which will include the additional number values, but probably not city names.

If there are too few fields the row it is filtered by returning True and an error message is printed.

Answer 2

I suggest you to filter before to optimize everything. Here the beginning of the path you should explore:

test_data = pd.DataFrame({'ID':['ID-10','ID-10','ID-20','ID-20','ID-30','ID-30','ID-40'],'id':[3,3,2,2,3,'GGG','GGG'],'cities':[['London','Manchester','London',1,1,1],['London','Manchester','London',1,1],['London','London',1,1],['London','London',1,1],['Madrid','Sevilla','Sevilla',1,1,1],['Madrid','Sevilla','Sevilla',1],['Madrid','Barçelona',1]]})

cities_to_filter = ['Sevilla', 'Manchester']
_condition1 = test_data.index.isin(test_data[test_data.id.str.isnumeric() != False][test_data[test_data.id.str.isnumeric() != False].id > 2].index)
test_data['results'] = np.where( _condition1,1,0)
test_data

OUTPUT:

And then you apply an 'any() in' for filtering the cities, but there are a lot of ways.