NLTK Python中的训练数据集

Question

I am working on Python NLTK tagging, and my input text is non hindi. In order to tokenize my input text it must first be trained.

My question is how to train the data?

I am having this line of code as suggested to me here on stackoverflow.

train_data = indian.tagged_sents('hindi.pos') 

*how about non-hindi data input.

Answer 1

The short answer is: Training a tagger requires a tagged corpus.

Assigning part of speech tags must be done according to some existing model. Unfortunately, unlike some problems like finding sentence boundaries, there is no way to choose them out of thin air. There are some experimental approaches that try to assign parts of speech using parallel texts and machine-translation alignment algorithms, but all real POS taggers must be trained on text that has been tagged already.

Evidently you don't have a tagged corpus for your unnamed language, so you'll need to find or create one if you want to build a tagger. Creating a tagged corpus is a major undertaking, since you'll need a lot of training materials to get any sort of decent performance. There may be ways to "bootstrap" a tagged corpus (put together a poor-quality tagger that will make it easier to retag the results by hand), but all that depends on your situation.