getting rid of proper nouns in a nested list python

Question

I'm trying to right a program that takes in a nested list, and returns a new list that takes out proper nouns.

Here is an example:

L = [['The', 'name', 'is', 'James'], ['Where', 'is', 'the', 'treasure'], ['Bond', 'cackled', 'insanely']]

I want to return:

['the', 'name', 'is', 'is', 'the', 'tresure', 'cackled', 'insanely']

Take note that 'where' is deleted. It is ok since it does not appear anywhere else in the nested list. Each nested list is a sentence. My approach to it is append every first element in the nested list to a newList. Then I compare to see if elements in the newList are in the nested list. I would lowercase the element's in the newList to check. I'm half way done with this program, but I'm running into an error when I try to remove the element from the newList at the end. Once i get the new updated list, I want to delete items from the nestedList that are in the newList. I'd lastly append all the items in the nested list to a newerList and lowercase them. That should do it.

If someone has a more efficient approach I'd gladly listen.

def lowerCaseFirst(L):
    newList = []
    for nestedList in L:
        newList.append(nestedList[0])
    print newList

    for firstWord in newList:
        sum = 0
        firstWord = firstWord.lower()
        for nestedList in L:
            for word in nestedList[1:]:
                if firstWord == word:
                    print "yes"

                    sum = sum + 1
            print newList
        if sum >= 1:
            firstWord = firstWord.upper()
            newList.remove(firstWord)
    return newList

Note this code is not finished due to the error in the second to last line

Here is with the newerList (updatedNewList):

def lowerCaseFirst(L):
    newList = []
    for nestedList in L:
        newList.append(nestedList[0])
    print newList
    updatedNewList = newList
    for firstWord in newList:
        sum = 0
        firstWord = firstWord.lower()
        for nestedList in L:
            for word in nestedList[1:]:
                if firstWord == word:
                    print "yes"

                    sum = sum + 1
            print newList
        if sum >= 1:
            firstWord = firstWord.upper()
            updatedNewList.remove(firstWord)
    return updatedNewList

error message:

Traceback (most recent call last):
  File "/Applications/WingIDE.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 1, in <module>
    # Used internally for debug sandbox under external interpreter
  File "/Applications/WingIDE.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 80, in lowerCaseFirst
ValueError: list.remove(x): x not in list

Answer 1

The error in your first function is because you try to remove an uppercased version of firstWord from newlist where there are no uppercase words (you see that from the printout). Remember that you store a upper/lowercased version of your words in a new variable, but you don't change the contents of the original list.

I still don't understand your approach. You want to do to things as you describe your task; 1) flatten the a lists of lists to a list of elements (always an interesting programming exercise) and 2) remove proper nouns from this list. This means that you have to decide what is a proper noun. You could do that rudimentarily (all non-starting capitalized words, or an exhaustive list), or you could use a POS tagger (see: Finding Proper Nouns using NLTK WordNet ). Unless I misunderstand your task completely, you needn't worry about the casing here.

The first task can be solved in many ways. Here is a nice way that illustrates well what actually happenes in the simple case where your list L is a list of lists (and not lists that can be infinitely nested):

def flatten(L):
  newList = []
  for sublist in L:
      for elm in sublist: 
          newList.append(elm)
  return newList

this function you could make into flattenAndFilter(L) by checking each element like this:

PN = ['James', 'Bond']

def flattenAndFilter(L):
  newList = []
  for sublist in L:
      for elm in sublist: 
          if not elm in PN:
              newList.append(elm)
  return newList

You might not have such a nice list of PNs, though, then you would have to expand on the checking, as for instance by parsing the sentence and checking the POS tags.