有没有人成功使用 Antlr v4 为 Hive 生成​​ javascript

Question

My aim is to parse SQL (specifically Hive) statements with javascript, preferably Nodejs. I started out with node-sql-parser which looked promising. However I found quite a few cases where the parser did not recognize valid SQL like several nested functions on a column in a select clause, and multiple AND clauses in SQL that had lots of joins, unions, etc. (I've logged as issue but it will take some time).

I decided to look at Antlr v4. I followed the getting started steps with Hive SQL grammar. ( https://github.com/apache/hive/blob/master/hplsql/src/main/antlr4/org/apache/hive/hplsql/Hplsql.g4 ); I generated parsers lexers and listeners using Antlr's generation for JavaSCript - all good so far. Then I tried a simple test as below:

const HplsqlLexer = require('./HplsqlLexer');
const HplsqlParser = require('./HplsqlParser');
const input = "select * from table_a"
var chars = new antlr4.InputStream(input);
var lexer = new HplsqlLexer.HplsqlLexer(chars);
var tokens = new antlr4.CommonTokenStream(lexer);
var parser = new HplsqlParser.HplsqlParser(tokens);
parser.buildParseTrees = true;
const tree = parser.program();

I believe "program()" is the entry point into the parser but I could be wrong. This gave me "ReferenceError: _input is not defined" at the parser.program() line. I questioned whether the Hplsql.g4 could be missing something but ruled that out. Then I looked at the generated code in HplsqlParser.js - I added var _input = "" at the top and reran; then it complained ablut LT is not defined. Feels like a rabbit hole.

Next steps include running the Java version of the Antlr parser, then Calcite. (hplsql.org is not what I am looking for).
node --version: v15.2.1. Any suggestions or pointers would be helpful.

Answer 1

As mentioned in the comments by kaby76: the grammar contains target specific (Java) code. You need to replace all Java code between { and }? with TypeScipt code.

For example, this Java code:

{!_input.LT(2).getText().equalsIgnoreCase("TRANSACTION")}?

can be rewritten into this:

{this._input.LT(2).text.toUpperCase() !== 'TRANSACTION'}?

(not tested!)

EDIT

I quickly did a global search and replace for pattern _input\\.LT\\((\\d+)\\).getText\\(\\)\\.equalsIgnoreCase\\("(\\w+)"\\) with replacement string (this._input.LT(\\1).text.toUpperCase() === '\\2') , which resulted in the following grammar:https://gist.github.com/bkiers/bb68b25ed03cf6c8ffae2709606d27a5

EDIT 2

I am surprised that Antlr even has a flag for -Dlanguage=JavaScript for parser generation. What's the point if it still is essentially Java?

The -Dlanguage=JavaScript makes sure to generate the lexer and parser classes in JavaScript. What it does not do, is rewrite semantic predicates , which are just copied "as-is". Note that it is always recommended to not use semantic predicates because of that, but move such target specific code into visitor or listener classes.