在Python中删除特殊字符

Question

I am trying to remove special characters from a log file. These are two example rows:

2016.04.03 23:54:28.257;:;213.210.213.316;:;PDL3_SGW2;:;5F6DBA-093E-0D4D9C-00000001-01;:;userId;:;;:;1000;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;250;:;;:;
2016.04.03 23:54:28.258;:;781.69.243.363;:;PDL3_SGW2;:;;:;userId;:;;:;1001;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;1;:;0x40001;:;Invalid credentials

This the output after removing the special characters:

2016.04.03  23  54  48.957  213.210.213.316  PDL3_SGW2  5F6DB03A    093E    0D414D9C    1   1   userId  1000    http    live.skysat.tv  cmdc    services    region  25351   lang    swe count   250 sort    2blogicalChannelNumber  101 0   250                                                                     

2016.04.03  23  54  48.958  781.69.243.363  PDL3_SGW2   userId  1001    http    live.skysat.tv  cmdc    services    region  25351   lang    swe count   250 sort    2blogicalChannelNumber  101 0   1   0xDC40001   Invalid credentials                                                                             

As you see in the second row of the ouput, the "userId" is positioned under column[6] instead of column [11]. Since data for column[06] to column[10] in the log file is missing.I want to handle this and write out the all the columns even when there is no data in the log file.

The output should be as following:

2016.04.03  23  54  48.957  213.210.213.316  PDL3_SGW2  5F6DB03A    093E    0D414D9C    1   1   userId  1000    http    live.skysat.tv  cmdc    services    region  25351   lang    swe count   250 sort    2blogicalChannelNumber  101 0   250                                                                     

2016.04.03  23  54  48.958  781.69.243.363  PDL3_SGW2                                           userId  1001    http    live.skysat.tv  cmdc    services    region  25351   lang    swe count   250 sort    2blogicalChannelNumber  101 0   1   0xDC40001   Invalid credentials                                                                             

This is my part of the code:

new_str = re.sub(r'[- - [ " / : ; & ? = % ~ + \n \]]', ' ', line)
text = new_str.rstrip().split()
writer.writerow(text)

Answer 1

This works for the two lines that you posted:

import re

lines = ["2016.04.03 23:54:28.257;:;213.210.213.316;:;PDL3_SGW2;:;5F6DBA-093E-0D4D9C-00000001-01;:;userId;:;;:;1000;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;250;:;;:;",
         "2016.04.03 23:54:28.258;:;781.69.243.363;:;PDL3_SGW2;:;;:;userId;:;;:;1001;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;1;:;0x40001;:;Invalid credentials"]

def adjust_columns(list_of_lines):
    widest = [max(len(el) for el in column) for column in zip(*list_of_lines)]
    return [ " ".join("{{:<{}s}}".format(widest[i]).format(e)
             for i,e in enumerate(line)) for line in list_of_lines ]

r = re.compile('[ /:;&?=%~+-]')
list_of_lines = [[r.split(el) for el in line.split(';:;')] for line in lines]
list_of_columns = [  all(len(el) == len(col[0]) for el in col)
                     and  adjust_columns(col)
                     or   [" ".join(el) for el in col]
                     for col in zip(*list_of_lines) ]
text = "\n".join(adjust_columns(list(zip(*list_of_columns))))
print(text)

This assumes that ;:; is always the delimiter for the fields. The code splits each line into the fields. Each field is then split again at the special characters. If each field in a column contains the same number of special characters, the subfields in that column are are adjusted for their width and joined by whitespace. The last step is to adjust the width of each column.

One problem could be, that you can't process the input line by line anymore, because you have to find the longest entry for each column.

If you do not need the subfields to be adjusted (like in your example), you can use this simpler code:

r = re.compile('[ /:;&?=%~+-]')
list_of_lines = [[" ".join(r.split(el)) for el in line.split(';:;')] for line in lines]
text = "\n".join(adjust_columns(list_of_lines))

Answer 2

>>> from pprint import pprint

Let's simulate the data file using a list of strings...

>>> lines = [
    '2016.04.03 23:54:28.257;:;213.210.213.316;:;PDL3_SGW2;:;5F6DBA-093E-0D4D9C-00000001-01;:;userId;:;;:;1000;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;250;:;;:;',
    '2016.04.03 23:54:28.258;:;781.69.243.363;:;PDL3_SGW2;:;;:;userId;:;;:;1001;:;http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber;:;101;:;0;:;1;:;0x40001;:;Invalid credentials']

From the official docs, you can use the string method S.split(sep) that returns a list of the words in S , using sep as the delimiter string (emphasis is mine).

In your case the delimiter is the string ';:;' , so you can do

>>> data = [line.split(';:;') for line in lines]

data is now a list of lists, each sublist contains empty strings for the missing fields in your file.

>>> pprint(data)
[['2016.04.03 23:54:28.257',
  '213.210.213.316',
  'PDL3_SGW2',
  '5F6DBA-093E-0D4D9C-00000001-01',
  'userId',
  '',
  '1000',
  'http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber',
  '101',
  '0',
  '250',
  '',
  ''],
 ['2016.04.03 23:54:28.258',
  '781.69.243.363',
  'PDL3_SGW2',
  '',
  'userId',
  '',
  '1001',
  'http://live.skysat.tv/cmdc/services?region=253&lang=swe&count=250&sort=%2blogicalChannelNumber',
  '101',
  '0',
  '1',
  '0x40001',
  'Invalid credentials']]

And you can loop over data and output each set of fields in the way you like best, eg,

>>> for record in data: output(record)
>>>

and that's all.

ps output() is a function that you have to define, according to your needs .