检测文本中的英语单词

Question

I have a dataset which is crawled but also contains entries that have a lot of junk in them.

Name: sdfsdfsdfsd
Location: asdfdgdfjkgdsfjs
Education: Science & Literature 

Currently its being stored in MySQL and Solr.
Is there any library that can look for English words in these fields so that i may eliminate the garbage values? I believe it would need a dictionary and the default unix dictionary in /usr/share/dict/ seems enough for this use-case.

Answer 1

with open('/usr/share/dict/words') as f:
    words = set(word.lower() for word in f.read().split()
                # Really short words aren't much of an indication
                if len(word) > 3)

def is_english(text):
    return bool(words.intersection(text.lower().split()))
    # or
    return any(word in words for word in text.lower().split())

print(is_english('usfdbg dsuyfbg cat'))
print(is_english('Science & Literature'))