How to tokenize a word containing punctuation using NLTK

Question

I have aPlainTextCorpusReader , and the text is job ads scraped from the web. I want to strip out the skills using NLTK.

But I have failed at the first hurdle as the reader is tokenizing the word 'C#' as 'C','#'.

I do want to filter out the noise such as stopwords and punctuation, so this is a problem.

How can I get around this?

Answer 1

答案是在使用语料库阅读器之前清理字符串，使用 replace 方法将 'C#' 替换为 'CSharp' 之类的内容，或者训练分词器。