Semantic parsing with NLTK

Question

I am trying to use NLTK for semantic parsing of spoken navigation commands such as "go to San Francisco", "give me directions to 123 Main Street", etc.

This could be done with a fairly simple CFG grammar such as

S -> COMMAND LOCATION
COMMAND -> "go to" | "give me directions to" | ...
LOCATION -> CITY | STREET | ...

The problem is that this involves non-atomic (more than one word-long) literals such as "go to", which NLTK doesn't seem to be set up for (correct me if I am wrong). The parsing task has tagging as a prerequisite, and all taggers seem to always tag individual words. So, my options seem to be:

a) Define a custom tagger that can assign non-syntactic tags to word sequences rather than individual words (eg, "go to": "COMMAND"). b) Use features to augment the grammar, eg, something like:

COMMAND -> VB[sem='go'] P[sem='to'] | ...

c) Use a chunker to extract sub-structures like COMMAND, then apply a parser to the result. Does NLTK allow chunker->parser cascading?

Some of these options seem convoluted (hacks). Is there a good way?

Answer 1

It seems like you want to identify imperatives.

This answer has looked into that and contains a solution similar to your option (a), but a bit different since it lets the tagger do most of the work. (b) indeed seems a bit hacky... but you're creating a pretty custom application, so it could work! I would do (c) the other way around - parsing and then chunking based on the CFG in (a).

Overall, however, as the other answer explains, there doesn't seem to be a perfect way to do this just yet .

You might also want to look at pattern.en . Their

mood() function tries to identify a parsed Sentence as indicative, imperative, conditional or subjunctive

Answer 2

For a related task (replication of a semantic parser of commercial transactions , I previously did something like (c). The biggest advantage of cascaded parser workflows is that the grammars can stay relatively simple (and fast), because they don't have to do global disambiguation. I guess it's the least hacky approach here because it's actually a valid strategy to design parser workflows.

As for (a), the custom tagger would require some kind of training data. If you can create or bootstrap that, that would be the most natural solution. For doing span annotations with a tagger, consider using the BIO (IOBES) schema as commonly used for named entity annotations, ie, You/B-COMMAND shall/I-COMMAND not/I-COMMAND pass/E-COMMAND !/O .

Answer 3

I would suggest you use the "rule-based matcher" that is available in spacy , which provides options as Token Matcher, Phrase Matcher, Entity Ruler. I have used Token Matcher specifying the pattern (go + to + place), and it worked very well.

token_pattern = [{"LEMMA": "go"}, {"POS": "ADP"}, {"POS": "PROPN"}]

This pattern would find "go to New York" or "going to New York" or "went with Mary", etc.

Also, if you are only trying to extract nouns like in your example "San Francisco", "123 Main Street", NER (Named Entity Recognition) might help.