Error in regex substring match in a list in python

Question

I have a list of lists as follows.

mylist = [["the", "and" "fresh milk", "a loaf of bread", "the butter"], ["an apple", "eggs", "oranges", "cup of tea"]]

Now I want to remove the stop words in mylist , so that my new list would be as follows.

mylist = [["fresh milk", "loaf bread", "butter"], ["apple", "eggs", "oranges", "cup tea"]]

My current code is as follows.

cleaned_mylist= []
stops = ['a', 'an', 'of', 'the']
pattern = re.compile(r'|'.join([r'(\s|\b){}\b'.format(x) for x in stops]))
for item in mylist:
    inner_list= []
    for words in item:
       inner_list.append(pattern.sub('', item).strip())
    cleaned_mylist.append(inner_list)

However, the code seems to be not working. Please help me.

Answer 1

You don't need to use regex in this example.

mylist = [["the", "and", "fresh milk", "a loaf of bread", "the butter"], ["an apple", "eggs", "oranges", "cup of tea"]]
expected = [["fresh milk", "loaf bread", "butter"], ["apple", "eggs", "oranges", "cup tea"]]

cleaned_mylist= []
stops = ['a', 'an', 'of', 'the', 'and']
for item in mylist:
    inner_list= []
    for sentence in item:
        out_sentence = []
        for word in sentence.split():
            if word not in stops:
                out_sentence.append(word)
        if len(out_sentence) > 0:
            inner_list += [' '.join(out_sentence)]
    cleaned_mylist.append(inner_list)

print expected == cleaned_mylist
# True

Answer 2

Your pattern is matching with Sublist (item) and not with the words

mylist = [["the", "and","fresh milk", "a loaf of bread", "the butter"], ["an apple", "eggs", "oranges", "cup of tea"]]
cleaned_mylist= []
stops = ['a', 'an', 'of', 'the','and']
pattern = re.compile(r'|'.join([r'(\s|\b){}\b'.format(x) for x in stops]))
for item in mylist:
    inner_list= []
    for words in item:
        if pattern.sub('', words).strip() != '':
            inner_list.append(pattern.sub('', words).strip())
    cleaned_mylist.append(inner_list)

Answer 3

Use if not

import re
mylist = [["the", "and", "fresh milk", "a loaf of bread", "the butter"], ["an apple", "eggs", "oranges", "cup of tea"]]
cleaned_mylist= []
stops = ['a', 'an', 'of', 'the','and']
pattern = '|'.join([r'\b{}\b\s?'.format(x) for x in stops])
for item in mylist:
    inner_list= []
    for words in item:
        words = re.sub(pattern,'',words)
        if(words !=  ""):
            inner_list.append(words)
    cleaned_mylist.append(inner_list)

print cleaned_mylist