performance issue my grammar and antlr4

Question

I'm new to using antlr 4 and have never written a grammar before. The grammar seems to work fine against short samples but I have a few longer examples that will run overnight without completion. I'm using the TestRig with my input file. I used JVisualVM and observed the code spending most of the time here:

Org.antlr.v4.runtime.atn.ArrayPredictionContext.equals(Object) (89.8%)
Org.antlr.v4.runtime.atn.SingletonPredictionContext.equals(Object) (7.4%)
Org.antlr.v4.runtime.misc.DoubleKeyMap.put(Object,Object,Object) (1.8%)
Org.antlr.v4.runtime.misc.DoubleKeyMap.get(Object, Object) (0.9%)

The problem appears to be related to the volume of statements of the following format.

reservedWord  = quotedstring

somewhere between 12 and 72 of these the problem occurs

i've given the jvm 2G of memory and it is using less than 1G. The heap grow and shrinks so I don't believe there is a memory leak.

Any advice on how to debug something like this?

I don't see a way to attach a file and the stack trace is 422 lines mostly repeating 1278 and 1325. i don't understand TestRig so I don't know what it is doing. Will get source code for antlr-4.0-complete.jar if I can find it.

at org.antlr.v4.runtime.atn.PredictionContext.hashCode(PredictionContext.java:121)  
at org.antlr.v4.runtime.atn.SingletonPredictionContext.equals(SingletonPredictionContext.java:97)   
at java.util.HashMap.getEntry(HashMap.java:349)     
at java.util.LinkedHashMap.get(LinkedHashMap.java:280)  
at org.antlr.v4.runtime.misc.DoubleKeyMap.get(DoubleKeyMap.java:60)     
at org.antlr.v4.runtime.atn.PredictionContext.mergeSingletons(PredictionContext.java:204)   
at org.antlr.v4.runtime.atn.PredictionContext.merge(PredictionContext.java:138)     
at org.antlr.v4.runtime.atn.ATNConfigSet.add(ATNConfigSet.java:354)     
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1293)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
<repeat a lot 1278 and 1325>

at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1261)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1261)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1261)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure_(ParserATNSimulator.java:1325)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closureCheckingStopState(ParserATNSimulator.java:1278)   
at org.antlr.v4.runtime.atn.ParserATNSimulator.closure(ParserATNSimulator.java:1216)    
at org.antlr.v4.runtime.atn.ParserATNSimulator.computeReachSet(ParserATNSimulator.java:988)     
at org.antlr.v4.runtime.atn.ParserATNSimulator.execATNWithFullContext(ParserATNSimulator.java:801)  
at org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:701)     
at org.antlr.v4.runtime.atn.ParserATNSimulator.predictATN(ParserATNSimulator.java:389)  
at org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:346)     
at ValidatingPolicyParser.showstatement(ValidatingPolicyParser.java:1023)   
at ValidatingPolicyParser.statement(ValidatingPolicyParser.java:685)    
at ValidatingPolicyParser.policy(ValidatingPolicyParser.java:246)   
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)    
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)    
at java.lang.reflect.Method.invoke(Method.java:597)     
at org.antlr.v4.runtime.misc.TestRig.process(TestRig.java:247)  
at org.antlr.v4.runtime.misc.TestRig.process(TestRig.java:209)  
at org.antlr.v4.runtime.misc.TestRig.main(TestRig.java:142)

I started working on a trimmed down grammar to use as a test case. This exercise did help me identify a couple of oddities that I removed from my grammar. This resolved the issue with my trimmed down grammar but when I restored the complete grammar the performance issue returned.

It really seems odd to me that increasing the number of occurrences of a simple compare increases the processing time non-linearly. A few instances of 'TOKEN = "SOMESTRING" OR TOKEN = "SOMEOTHERSTING"' processes relatively quickly but if I have 75 of these compares the processing time becomes way too much.

I can't determine if antlr4 source is available somewhere or not. Would be nice to step through the code to see what is happening.

Guess I'll repeat the process and see what it yields.

Answer 1

Those timing figures are useless. They tell you epsilon.

Instead, just interrupt it. You will catch it in the act. Examine the call stack and, if necessary, data variables. If you understand what it's doing at that point in time, you'll know what the problem is.

More than you ever wanted to know about that.

Answer 2

well I found the problem in my grammar. took a lot of trial and error. I wonder if recursion is being used in antlr4?