Split a string on multiple characters in python

Question

I am trying to split a string on multiple characters in python just like I am doing in Java like this:

private static final String SPECIAL_CHARACTERS_REGEX = "[ :;'?=()!\\[\\]-]+|(?<=\\d)(?=\\D)";
String rawMessage = "let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]";
String[] tokens = rawMessage.split(SPECIAL_CHARACTERS_REGEX);
System.out.println(Arrays.toString(tokens));

Here is the working demo with the correct output: Working Demo

I am trying to do exactly the same in python, but when I am doing that it would not tokenize at all if I just add the 'single quotes' character in the regex. How do I create the same resultant parse results from python as from Java program above?

This:

import re
tokens = re.split(' \.', line);
print tokens

For line:

"let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]"

Gives:

["let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]";]

When I was it to do this:

[let, s, meet, tomorrow, at, 9, 30, p, 7, 8, pm, i, you, go, no, Go, to, do]

Answer 1

Here's an alternative that finds rather than splits:

>>> s = "let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]"
>>> re.findall(r'\d+|[A-Za-z]+', s)
['let', 's', 'meet', 'tomorrow', 'at', '9', '30', 'p', '7', '8', 'pm', 'i', 'you', 'go', 'no', 'Go', 'to', 'do']

If it is ok to keep letters and numbers together use '[0-9A-Za-z]+' . For letters, numbers, and underscore use r'\\w+' .

Answer 2

Use the same regular expression you used in Java:

line = "let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]"
tokens = re.split("[ :;'?=()!\\[\\]-]+|(?<=\\d)(?=\\D)", line)
tokens = [token for token in tokens if len(token) != 0] # remove empty strings!
print(tokens)
# ['let', 's', 'meet', 'tomorrow', 'at', '9', '30p', '7', '8pm', 'i', 'you', 'go', 'no', 'Go', 'to', 'do']

Answer 3

Use the following code

>>> chars = "[:;'?=()!\-]+<" #Characters to remove
>>> sentence = "let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]" #Sentence
>>> for k in sentence: #Loops over everything in the sentence
...     if k in chars: #Checks if the variable is one we want to remove
...             sentence = sentence.replace(k, ' ') #If it is, it replaces it
...
>>> sentence = sentence.replace('p', ' p').replace('pm', ' pm').split() #Adds a space before the 'p' and the 'pm', and then splits it the way we want to
>>> sentence
['let', 's', 'meet', 'tomorrow', 'at', '9', '30', 'p', '7', '8', 'pm', 'i', 'you', 'go', 'no', 'Go', 'to', 'do']

If you want to use regex :

line = "let's meet tomorrow at 9:30p? 7-8pm? i=you go (no Go!) [to do !]"
tokens = re.split("[ :;'?=()!\\[\\]-]+|(?<=\\d)(?=\\D)", line)
tokens = [token for token in tokens if len(token) != 0]
tokens = tokens.replace('p', ' p').replace('pm', ' pm').split()
print(tokens)
#['let', 's', 'meet', 'tomorrow', 'at', '9', '30', 'p', '7', '8', 'pm', 'i', 'you', 'go', 'no', 'Go', 'to', 'do']

Answer 4

That split regex in Java should have worked the same in Python.
Its probably a bug. The confusion would probably be the overlap
between \\D and [ :;'?=()!\\[\\]-] , and how it handles that (bug~).

You could try to solve it by putting (?<=\\d)(?=\\D) first, but it
has to be coerced to do that.

This regex here forces it to do that. Is this a workaround?
I don't know, don't have python to test with. But, it works in Perl.

Coerced regex -

 #  (?<=\d)(?:[ :;'?=()!\[\]-]+|(?=\D))|(?<!\d|[ :;'?=()!\[\]-])[ :;'?=()!\[\]-]+

    (?<= \d )
    (?:
         [ :;'?=()!\[\]-]+ 
      |  (?= \D )
    )
 |  
    (?<! \d | [ :;'?=()!\[\]-] )
    [ :;'?=()!\[\]-]+