使用单词列表python删除字典中的值

Question

Let's say I have a list of words

 nottastyfruits = ['grape', 'orange', 'durian', 'pear']

 fruitGroup = {'001': ['grape','apple', 'jackfruit', 'orange', 'Longan'],
               '002': ['apple', 'watermelon', 'pear']}

I want to go through all the keys in the dictionary and remove the words from nottastyfruits list.

My current code is

finalfruits = {}
for key, value in fruitGroup.items():
    fruits = []
    for fruit in value:
        if fruit not in nottastyfruits:
            fruits.append(fruit)
    finalfruits[key] = (fruits)

This takes so long to run when you have a large data text such as large text preprocessing. Is there a more efficient and faster way to do this?

Thank you for you time

Answer 1

You should make a set out of your fruitlist to speedup the lookups, then use a dictionary comprehension:

nottastyfruits = set(['grape', 'orange', 'durian', 'pear'])

fruitGroup = {'001': ['grape','apple', 'jackfruit', 'orange', 'Longan'],
           '002': ['apple', 'watermelon', 'pear']}

print {k: [i for i in v if i not in nottastyfruits] for k, v in fruitGroup.iteritems()}

>>> {'002': ['apple', 'watermelon'], '001': ['apple', 'jackfruit', 'Longan']}

Answer 2

Making it flat by using a dictionary comprehension will remove the overhead of the for loop.

Making nottastyfruits a set will decrease lookup time:

nottastyfruits  = set(nottastyfruits)
finalfruits = {k: [f for f in v if f not in nottastyfruits] for k, v in fruitGroup.items()}

Answer 3

One low-hanging fruit, if you will, is to make nottastyfruits a set . Also, you can use comprehensions to squeeze some performance out.

In [1]: fruitGroup = {'001': ['grape','apple', 'jackfruit', 'orange', 'Longan'],
   ...:                '002': ['apple', 'watermelon', 'pear']
   ...:               }

In [2]: nottastyfruit = {'grape', 'orange', 'durian', 'pear'}

In [3]: finalfruits = {k:[f for f in v if f not in nottastyfruit] for k,v in fruitGroup.items()}

In [4]: finalfruits
Out[4]: {'001': ['apple', 'jackfruit', 'Longan'], '002': ['apple', 'watermelon']}

Answer 4

Since both nottastyfruits and lists in the dictionary are flat lists, you can use sets to get the difference between the two.

nottastyfruits = set(['orange', 'pear', 'grape', 'durian'])
fruitGroup = {'001': ['grape','apple', 'jackfruit', 'orange', 'Longan'], '002': ['apple', 'watermelon', 'pear'] }

for key, value in fruitGroup.iteritems():
    fruitGroup[key] = list(set(value).difference(nottastyfruits))

print fruitGroup # Prints "{'002': ['watermelon', 'apple'], '001': ['jackfruit', 'apple', 'Longan']}"

Answer 5

Below is a benchmark of differents proposed solutions plus a solution based on the filter() function:

from timeit import timeit


nottastyfruits = ['grape', 'orange', 'durian', 'pear']

fruitGroup = {'001': ['grape','apple', 'jackfruit', 'orange', 'Longan'],
              '002': ['apple', 'watermelon', 'pear']}


def fruit_filter_original(fruit_groups, not_tasty_fruits):
    final_fruits = {}
    for key, value in fruit_groups.items():
        fruits = []
        for fruit in value:
            if fruit not in not_tasty_fruits:
                fruits.append(fruit)
        final_fruits[key] = (fruits)
    return final_fruits


def fruit_filter_comprehension(fruit_groups, not_tasty_fruits):
    return {group: [fruit for fruit in fruits
                         if fruit not in not_tasty_fruits]
            for group, fruits in fruit_groups.items()}


def fruit_filter_set_comprehension(fruit_groups, not_tasty_fruits):
    not_tasty_fruits = set(not_tasty_fruits)
    return {group: [fruit for fruit in fruits
                         if fruit not in not_tasty_fruits]
            for group, fruits in fruit_groups.items()}


def fruit_filter_set(fruit_groups, not_tasty_fruits):
    return {group: list(set(fruits).difference(not_tasty_fruits))
            for group, fruits in fruit_groups.items()}


def fruit_filter_filter(fruit_groups, not_tasty_fruits):
    return {group: filter(lambda fruit: fruit not in not_tasty_fruits, fruits)
            for group, fruits in fruit_groups.items()}


print(fruit_filter_original(fruitGroup, nottastyfruits))
print(fruit_filter_comprehension(fruitGroup, nottastyfruits))
print(fruit_filter_set_comprehension(fruitGroup, nottastyfruits))
print(fruit_filter_set(fruitGroup, nottastyfruits))
print(fruit_filter_filter(fruitGroup, nottastyfruits))


print(timeit("fruit_filter_original(fruitGroup, nottastyfruits)", number=100000,
      setup="from __main__ import fruit_filter_original, fruitGroup, nottastyfruits"))
print(timeit("fruit_filter_comprehension(fruitGroup, nottastyfruits)", number=100000,
      setup="from __main__ import fruit_filter_comprehension, fruitGroup, nottastyfruits"))
print(timeit("fruit_filter_set_comprehension(fruitGroup, nottastyfruits)", number=100000,
      setup="from __main__ import fruit_filter_set_comprehension, fruitGroup, nottastyfruits"))
print(timeit("fruit_filter_set(fruitGroup, nottastyfruits)", number=100000,
      setup="from __main__ import fruit_filter_set, fruitGroup, nottastyfruits"))
print(timeit("fruit_filter_filter(fruitGroup, nottastyfruits)", number=100000,
      setup="from __main__ import fruit_filter_filter, fruitGroup, nottastyfruits"))

We can see that all solutions are NOT equal in term of performance:

{'001': ['apple', 'jackfruit', 'Longan'], '002': ['apple', 'watermelon']}
{'001': ['apple', 'jackfruit', 'Longan'], '002': ['apple', 'watermelon']}
{'001': ['apple', 'jackfruit', 'Longan'], '002': ['apple', 'watermelon']}
{'001': ['jackfruit', 'apple', 'Longan'], '002': ['watermelon', 'apple']}
{'001': ['apple', 'jackfruit', 'Longan'], '002': ['apple', 'watermelon']}
2.57386991159  # fruit_filter_original
2.36822144247  # fruit_filter_comprehension
2.46125930873  # fruit_filter_set_comprehension
4.09036626702  # fruit_filter_set
3.76554637862  # fruit_filter_filter

The comprehension based solution is the better but it is not a very significant improvement (with the given data at least) compared to the original code. The set comprehension solution is also a small improvement. The solutions based on filter function and set difference are quite slow...

Conclusion : If you are looking for performance, the solutions from Moses Koledoye and juanpa.arrivillaga seem to be better. However, those results could be different with bigger data, so it could be a good idea to do the test with real data.