How to tokenize a string (which has data about mathematical calculations and floating point numbers)?

Question

I'm trying to tokenize a string (which has data about mathematical calculations) and create a list.

for example,

a = "(3.43 + 2^2 / 4)"

function(a) => ['(', '3.43', '+', '2', '^', '2', '/', '4']

I don't want to use external imports (like nltk).

The problem I'm facing is keeping the floating point numbers intact.

I've been scratching my head for hours and have made 2 functions, but the problem occurs when it confronts floating point numbers.

Here is what I've done:

a = "(3.43 + 2^2 / 4)"
tokens = []

for x in range(1, len(a)-1):
no = []

if a[x] == ".":
    y = x
    no.append(".")

    while is_int(a[y-1]):
        no.insert(0, a[y-1])
        y -= 1

    y = x

    while is_int(a[y+1]):
        no.extend(a[y+1])
        y += 1

    token = "".join(no)
    no = []
    tokens.append(token)

else:
    tokens.append(a[x])

print(tokens)

OUTPUT:

['3', '3.43', '4', '3', ' ', '+', ' ', '2', '^', '2', ' ', '/', ' ', '4']

Answer 1

You could use Python's own tokenizer, which is part of the standard API:

from tokenize import tokenize
from io import BytesIO

source = "(3.43 + 2^2 / 4)"
tokens = tokenize(BytesIO(source.encode('utf-8')).readline)
non_empty = [t for t in tokens if t.line != '']

for token in non_empty:
    print(token.string)

which will print:

(
3.43
+
2
^
2
/
4
)

More info: https://docs.python.org/3/library/tokenize.html

Answer 2

Try this

a = "(3.43 + 2^2 / 4)"
tokens = []
no = ""

for x in range(0, len(a)):
    # Skip spaces
    if a[x] == " ":
        pass
    # Concatenate digits or '.' to number
    elif a[x].isdigit() or (a[x] == "."):
        no += a[x]
    # Other token: append number if any, and then token
    else:
        if no != "":
            tokens.append(no)
        tokens.append(a[x])
        no = ""

print(tokens)

Output:

['(', '3.43', '+', '2', '^', '2', '/', '4', ')']

Note, this won't handle operators that are more than one character, such as ==, **, +=

Answer 3

Notice also that your code won't work when you have more than 1 digits numbers in your expression. But you can try this:

a = "(3.43 + 22^222 / 4)"
list_a = a[1:-1].split()  # remove first and last paranthesis and split by expression
tokens = []
for val in list_a:
    if '.' in val:  # if number has '.' then convert it to float number.
        tokens.append(float(val))
    elif val.isdigit():  # if it is number then add it to tokens
        tokens.append(val)
    elif len(val)==1:  # if it is math expression then add it to tokens
        tokens.append(val)
    else:  # problem is when you have an expression like: "2^2" - we have to go char by char
        no = []
        for k in val:
            if k.isdigit():
                no.append(k)
            else:
                tokens.append(''.join(no))
                tokens.append(k)
                no = []
        tokens.append(''.join(no))

print(tokens)