如何将数学表达式拆分为标记？ （字符串-> [String]）

Question

I know my question sounds generic and I didn't really know how to google for such a thing, so I'll provide examples of what I mean:

ghci> splitTok "12 + 564 * (5 - 38) / 223"
["12", "+", "564", "*", "(", "5", "-", "38", ")", "/", "223"]
ghci> splitTok " 5+2-42  *    (46/5 )"
["5", "+", "2", "-", "42", "*", "(", "46", "/", "5", ")"]

How could something like that be implemented? Or maybe there is a function in Prelude that does this type of thing?

Answer 1

It seems just to group the digits together and removes the space. Try

import Data.Char (isSpace, isDigit)
import Data.List (groupBy)

splitTok = groupBy (\x y->isDigit x && isDigit y) . filter (not . isSpace)

Answer 2

Okay, I came up with an ugly hack that does the job. (Unfortunately introduces a bug)

import Data.Char (isSpace, isDigit)

strip :: String -> String
strip = filter (not . isSpace)

addSpacing :: String -> String
addSpacing [a] = [a]
addSpacing (x:y:cs)
    | isDigit x && isDigit y = x : addSpacing rest
    | otherwise = x : ' ' : addSpacing rest
  where rest = y : cs

splitTok :: String -> [String]
splitTok = words . addSpacing . strip

It fails in producing a correct string of tokens in this example:

ghci> splitTok "125 +   12 62 - 12  *( 51/  3)        "
["125","+","1262","-","12","*","(","51","/","3",")"]

For most expressions it works fine though:

ghci> splitTok "4123-36522+12"
["4123","-","36522","+","12"]
ghci> splitTok "124 *(12 -(4+5*(331/7)))"
["124","*","(","12","-","(","4","+","5","*","(","331","/","7",")",")",")"]