在Java的Stanford NLP解析器中设置选项

Question

I am trying to use the Stanford NLP Parser to parse POS tagged data. Since my data is already tagged and tokenized I am trying to use the setOptionFlags() method to inform the parser about this like,

LexicalizedParser lp = LexicalizedParser.loadModel("edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz");
lp.setOptionFlags(new String[]{"-sentences", "newline", "-tokenized", "-tagSeparator", "_", "-tokenizerFactory", "edu.stanford.nlp.process.WhitespaceTokenizer", "-tokenizerMethod", "newCoreLabelTokenizerFactory"});

However, I keep getting an exception,

Exception in thread "main" java.lang.IllegalArgumentException: Unknown option: -sentences

I have searched online through the Javadocs provided and this is the way that it is done in their examples. Please help!

Answer 1

The options for tokenization, tag separator, etc. are not options for the parser, sensu stricto, but for the DocumentPreprocessor that is used to build input to the parser in the main method of LexicalizedParser . For the actual parser, the input is a list of tokens, and these are parsed. Hence, you can't give these options as parser options with setOptions() .

If you've got a List of tokens, you can put them straight into the parser with this method in LexicalizedParser : public Tree parse(List<? extends HasWord> lst) . If the items in the list implement HasTag (eg, a TaggedWord or a CoreLabel ) and have a non-null tag, then that will be used by the parser in parsing the sentence.

If you want to use a DocumentPreprocessor to split up text with tokenized tagged words, then you need to create a DocumentPreprocessor and then to set things up (a bit manually, sorry) with the methods like setTagDelimiter(String s) .