Shell Input Independent of Spaces Implementation

Question

I recently had a programming class where we implemented a shell in Java. One of the requirements was to ensure that parameters could be read from terminal regardless of spaces and tabs etc between them unless within quotes where everything within quotes is taken as is.

To solve this I wrote a regex and used streams to get the results in an array for further processing.

But now while preparing for a systems programming course I realized there must be a simpler way to do this? How is this implemented in a typical shell like bash?

Is it just reading the stream character by character and skipping when it runs into quotes until it finds a matching one?

Answer 1

For bash (and probably any other modern shell), it is much more complex than that. See this function in bash's source code that is used to parse a matching pair of characters (quotes, curly braces etc.). It is pretty complex as there are many different types of quotes, parentheses and braces ( ' , { , ( , " , ' , ...) and a lot of edge cases are involved. For example, in this case, skipping characters until you see another quotes wouldn't work because things can be nested:

echo "`echo "hello"`"

I don't know about the requirements of the shell program you implemented in your class, but if it didn't include such nested constructs, then I believe a simple approach like you mentioned could be used.

Answer 2

For complex grammars (and bash tokens can be complex), better to use parser/generator tools, instead of implementing the logic from scratch. Using RE, possible to get some grammars covered, but it's unlikely to cover complex set of rules.

Depending on the constraints (programming language, etc.), consider two options:

• Using flex/bison for token parsing and grammar parsing, OR
• Using a scripting engine (Python, Perl, JavaScript), which has both RE and strong string processing capabilities.