How can I use python iterable to perform flexible search generators?

Question

EDIT: Turns out this isn't much of a question. I managed to get this working as I more or less typed this in.

I've got an online csv table database that I query with a cgi script. The script can take several optional filter arguments, but adding a new filter means I have to double my if statements.

Ex:

records = list()
with open (csv_datafile, 'r') as data: 
    data_reader = csv.DictReader(data, fieldnames=DATA_FIELDS)
    if product is None:
        records.extend(record for record in data_reader if id_start < record['id'] <= id_end)
    else:
        records.extend(record for record in data_reader if id_start < record['id'] <= id_end and record['code'] == product)

You can see, that if I add another search criteria, this pattern falls apart because I'd have to nest redundant if's in each of the above if clauses.

Is there a way I use an iterator to do this? My thinking is along the lines of:

def data_search(iterator, **search)
    for item in iterable:
        yield_me=True
        for key in item.viewkeys() & search:
            if item[key] is not None:
                if item[key] != search[key]:
                    yeild_me=False
        if yield_me:
            yield item

Then the code to integrate would be something like:

search = dict()
if product_search:
    search['code'] = product

if vendor_search:
    search['vendor'] = vendor

records = list()
with open (csv_datafile, 'r') as data: 
    data_reader = csv.DictReader(data, fieldnames=DATA_FIELDS)
    data_reader = data_search(data_reader, **search)
    records.extend(record for record in data_reader if id_start < record['id'] <= id_end)

Should I continue on this tack, or is there a better way?

Answer 1

I don't think you need a generator for this. I think you're on the right track with building up your search conditions (although if you can get them passed in as name/value pairs in a dictionary that's easier to work with), but enforcing it can be simpler.

with open(csv_datafile, 'r') as data:
    data_reader = csv.DictReader(data, fieldnames=DATA_FIELDS)
    records = [record for record in data_reader if all(record.get(filter_key) == filter_value for (filter_key, filter_value) in search.iteritems()) and id_start < record['id'] <= id_end]

There are different expressions possible using filter if you prefer, and I might make a passes_search_filters helper function for legibility.