从列表中删除数字（如果未包含在其他列表的子字符串中）

Question

Here's my situation:

I have one list of product names such as:
BLUEAPPLE, GREENBUTTON20, 400100DUCK20 (len = 9000)
and a list of official item names such as:
BLUEAPPLE, GREENBUTTON, 100DUCK . (len = 2700)

As I'll be applying fuzzy string matching to product - items, I want to strip away the unnecessary numbers from the product names -- but keep numbers that are represented in official item names.

I came up with a solution, but the issue is it works very slowly.

def remove_nums(product):
    if bool(re.search('\d'), product):
        for item in item_nums_list:
            if item in product_name:
                substrings = [u for x in product_name.split(item) for u in (x, item)][:-1]
                no_num_list = [re.sub('(\d+)', '', substring) if substring not in item else substring for substring in substrings]
                return ''.join(no_num_list)
        return re.sub('(\d+)', '', product)
    else:
        return product

Example:

product_name = '400100DUCK20'
item = '100DUCK'
substrings = ['400','100DUCK','20']
no_num_list = ['','100OG','']
returns '100DUCK'

This function is mapped so that it's looping over every product in the product list.

I've been trying to figure out a way to use lambdas here, maps, applys, etc, but can't quite wrap my head around it. What would be the most efficient way to accomplish what I am trying to do, either with straight lists, or in pandas? Alternatively, I'm getting these item and product lists from a postgres database, so if you think it'd be faster to do in psql I'd go that route.

Answer 1

difflib.get_close_matches() will at least help clean up your code and will probably run faster.

import difflib
p_names = ['BLUEAPPLE', 'GREENBUTTON20', '400100DUCK20']
i_names = ['BLUEAPPLE', 'GREENBUTTON', '100DUCK']
for p in p_names:
    print(difflib.get_close_matches(p, i_names))

>>> 
['BLUEAPPLE']
['GREENBUTTON']
['100DUCK']
>>> 

There are still going to be a lot of comparisons taking place, it has to match every string in p_names to every string in i_names.

---

Similar to your approach using a regular expressions to find a match:

import re
for p in p_names:
    for i in i_names:
        if re.search(i, p):
            print(i)
            # stop looking
            break

Answer 2

Try this:

def remove_nums(product):
    if re.search('\d', product):
        for item in item_nums_list:
            if item in product:
                return item
        return re.sub('(\d+)', '', product)
else:
    return product

Also, make sure you are using the normal python interpreter. IPython and other interpreters with debugging features are a LOT slower than the regular interpreter.

You might want to consider doing some set operations first though. Here's a little example:

product_set = set(product_list)
item_number_set = set(item_number_list)

# these are the ones that match straight away
product_matches = product_set & item_number_set

# now we can search through the substrings of ones that don't match
non_matches = product_set - item_number_set
for product in non_matches:
    for item_number in item_number_set:
        if item_number in product:
            product_matches.add(product)
            break

# product_matches is now a set of all unique codes contained in both lists by "fuzzy match"
print(product_matches)

You kind of lose the order in which they appeared, but maybe you can find a way to modify this for your use.