计算文件中的单词时，从单词的开头和结尾删除特殊字符

Question

I need to count words in a huge text file but before that, I have to clean the file of special characters in a specific way.

For example -

;xyz        -->      xyz      
xyz:        -->     xyz          
xyz!)       -->     xyz!

I am using flatMap() to split all the words on space. And then I am trying to remove the special characters which is not working. Please help!

Here is the code I am using ---

The characters to remove are - : ; ! ? ( ) .

   >>> input = sc.textFile("file:///home/<...>/Downloads/file.txt")
   >>> input2 = input.flatMap(lambda x: x.split())
   >>> def remove(x):
           if x.endsWith(':'):
                x.replace(':','')
                return x
           elif x.endsWith('.'):
               x.replace('.','')
               return x

. .

      >>> input3 = input2.map(lambda x: remove(x))

Answer 1

Use re.sub

re.sub(r'(?<!\S)[^\s\w]+|[^\s\w]+(?!\S)', '', f.read())

DEMO

Answer 2

You can write a function that sees if a character is valid, then use filter() :

def is_valid(char):
    return char.isalpha() or char in "!,." # Whatever extras you want to include

new_string = ''.join(filter(is_valid, old_string)) # No need to ''.join() in Python 2

Answer 3

Try getting help of regex:

import re

with open('input.txt','r') as fp:
    rx = "[;:\)]+"
    for line in fp:
        data = re.sub(rx, "", line.strip())
        print(data)

Code above will read file line by line and emit sanitized content. Depending on content of file it will print:

xyz
xyz
xyz!

Answer 4

This is the code that worked for me-
def removefromstart(x):
... for i in [':','!','?','.',')','(',';',',']:
... if x.startswith(i):
... token = x.replace(i,'')
... return token
... return x
...

 def removefromend(x): ... for i in [':','!','?','.',')','(',';',',']: ... if x.endswith(i): ... token = x.replace(i,'') ... return token ... return x