使用Spacy进行令牌化-如何获取左右令牌

Question

I am using Spacy for text tokenization and getting stuck with it:

import spacy
nlp = spacy.load("en_core_web_sm")
mytext = "This is some sentence that spacy will not appreciate"
doc = nlp(mytext)

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_, token.is_alpha, token.is_stop)

returns something that seems to me to say that tokenisation was succesful:

This this DET DT nsubj Xxxx True False 
is be VERB VBZ ROOT xx True True 
some some DET DT det xxxx True True 
sentence sentence NOUN NN attr xxxx True False 
that that ADP IN mark xxxx True True 
spacy spacy NOUN NN nsubj xxxx True False 
will will VERB MD aux xxxx True True 
not not ADV RB neg xxx True True 
appreciate appreciate VERB VB ccomp xxxx True False

but on the other hand

[token.text for token in doc[2].lefts]

returns an empty list. Is there a bug in lefts/rights ?

Beginner at natural language processing, hope I am not falling into a conceptual trap. Using Spacy v'2.0.4'.

Answer 1

This is what the dependencies of that sentence look like:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp(u"This is some sentence that spacy will not appreciate")
for token in doc:
    print(token.text, token.dep_, token.head.text, token.head.pos_,
            [child for child in token.children])

This nsubj is VERB []
is ROOT is VERB [This, sentence]
some det sentence NOUN []
sentence attr is VERB [some, appreciate]
that mark appreciate VERB []
spacy nsubj appreciate VERB []
will aux appreciate VERB []
not neg appreciate VERB []
appreciate relcl sentence NOUN [that, spacy, will, not]

So we see that doc[2] ("some") has an empty child vector. However "is" ( doc[1] ) does not. If we instead run...

print([token.text for token in doc[1].lefts])  
print([token.text for token in doc[1].rights])  

we get...

['This']
['sentence']

The functions you are using navigate the dependency tree, not the document, hence why you are getting empty results for some words.

If you just want previous and following tokens, you can just do something like...

for ix, token in enumerate(doc):
    if ix == 0:
        print('Previous: %s, Current: %s, Next: %s' % ('', doc[ix], doc[ix + 1]))
    elif ix == (len(doc) - 1):
        print('Previous: %s, Current: %s, Next: %s' % (doc[ix - 1], doc[ix], ''))
    else:
        print('Previous: %s, Current: %s, Next: %s' % (doc[ix - 1], doc[ix], doc[ix + 1]))

or...

for ix, token in enumerate(doc):
    print('Previous: %s' % doc[:ix])
    print('Current: %s' % doc[ix])
    print('Following: %s' % doc[ix:])