Java Simple Lexer程序

Question

I created a simple lexer program in Java which prompts the user for a string and displays the lexemes in that String. However, when I enter a value, if left and/or right parentheses is included in the prompt, after the left or right parentheses a null character is added which it identifies as an identifier by the program.

Also, if I don't include left and right parentheses in the user prompted String, the last character in the String is not evaluated as a lexeme.

Here is my code :

import javax.swing.JOptionPane;

public class Append
{
  public static void main (String [] args)
  {
    String str = JOptionPane.showInputDialog("Enter string : ");
    char [] arr = str.toCharArray();

    JOptionPane.showMessageDialog(null,arr.length);

    determineLexemes(arr);

   }

  public static void determineLexemes(char [] arr)
  {
    int j = 0;

    String [] arrayString = new String [1000];

    String strTwo = "";

    System.out.println("Symbol Table");

    System.out.println("Lexeme\t\tToken");

    for(int i = 0; i < arr.length; i++)
    {

       if(arr[i] == '+')
            {
                System.out.println("+ \t\t ADD_OP");
            }

       if(arr[i] == '-')
            {
                System.out.println("- \t\t SUB_OP");
            }

       if(arr[i] == '*')
            {
                System.out.println("* \t\t MULT_OP");
            }

       if(arr[i] == '/')
            {
                System.out.println("/ \t\t DIV_OP");
            }

       if(arr[i] == '(')
            {
                System.out.println("( \t\t LEFT_PAREN");
            }

       if(arr[i] == ')')
            {
                System.out.println(") \t\t RIGHT_PAREN");
            }

       if(arr[i] == '=')
            {
                System.out.println("= \t\t EQUAL_OP");
            }

       if(Character.isLetter(arr[i]) || Character.isDigit(arr[i]))
        {
            strTwo += arr[i];
        }

       if(!Character.isLetter(arr[i]) && !Character.isDigit(arr[i]))
        {
            if(!(Character.isWhitespace(arr[i])))
            {
                arrayString[j] = strTwo;
                System.out.println(arrayString[j] + "\t\t" + "IDENTIFIER");
                strTwo = "";
                j++;

            }
        }
    }


 }
}

Any help to resolve the problem is appreciated.

Answer 1

The problem is that you do not maintain state in your lexer. Recognizing a regular language can be done with a finite automaton, which is a simple mechanism that keeps track of its state (and may maintain a buffer for accumulating longer lexemes).

So, initially you should set the state to S0, and each operator and parentheses is recognized, and you stay in state S0. For a letter, you enter SI, and remain, while recognizing in SI for more letters and digits. An operator terminates SI, and emits the operator and returns to S0. - Recognizing a digit in S0, enters SN, and you handle this in a way similar to SI.

enum State { S0, IDENTIFIER, NUMBER } 

State state = State.S0;
for(int i = 0; i < arr.length; i++) {
switch( state ){
case S0:
    switch(arr[i]){
    case '+':
        System.out.println("+ \t\t ADD_OP");
        break;
    //...
    default:
        if(Character.isLetter(arr[i])){
            strTwo = ""; strTwo += arr[i];
            state = State.IDENTIFIER;
        }
        if(Character.isDigit(arr[i])){
            strTwo = ""; strTwo += arr[i];
            state = State.NUMBER;
        }
        break;
    }
case IDENTIFIER:
    if(Character.isLetter(arr[i]) || Character.isDigit(arr[i])){
        strTwo += arr[i];
    } else {
        System.out.println(strTwo + "\t\t" + "IDENTIFIER");
        i--;
        state = State.S0;
    }
        break;
case NUMBER:
    if(Character.isDigit(arr[i])){
        strTwo += arr[i];
    } else {
         System.out.println(strTwo + "\t\t" + "NUMBER");
         i--;
         State = State.S0;      
    }
    break;
}

Something is missing here: Handling a number or identifier at the end of the input string. This can be determined by examining variable state and using the contents of strTwo.