如何使用 SpaCy 更改自定义 NER model 再训练的训练数据格式？

Question

I am working on this problem where the text data is in the a document file and the resulting 5 tags are in a csv file. So to train spaCy NER model, we have to tag dtaa something like:

TRAIN_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]}),
]

But my data is in csv file like:

I wrote a function which will search the first occurrences of col query in the text and add the length. Something like:

train_data = []
for i,index in enumerate(df.index.tolist()):
    row_data = df.iloc[i,:].values.tolist()
    entities = {"entities":[]}
    for file in dir_files:
        if file.split('.')[0] == row_data[0]:
            text = preprocess(textract.process("./Training_data/"+file))
            
            for j,entry in enumerate(row_data[1:]):
                
                if not pd.isna(entry):
                    if isinstance(entry,str): # takes care of null values
                        entities['entities'].append((text.find(str(entry).strip()),len(str(entry)),ent_names[j]))

and the result is

{'entities': [(-1, 7, 'Aggrement Value'),
  (-1, 10, 'Aggrement Start Date'),
  (-1, 10, 'Aggrement End Date'),
  (-1, 4, 'Renewal Notice (Days)'),
  (124, 22, 'Party One'),
  (540, 45, 'Party Two')]}

It is giving me decent results for the STRING but I have a huge problem for date as the are in format 12.08.2018 and price which is format 6000.00 . I can't compare directly so I have to change the price str(int(price)) and then match. It'll work BUT the date is never in the format given in CSV. It's spmething like 1stDAY OF SEPTEMBER 2018 TWO THOUSAND EIGHTEEN . How am I supposed to tag that one in format?

I tried using Spacy's inbuilt NER so that I could figure out but it is not giving me good results.

nlp = spacy.load('en_core_web_sm')
doc = nlp(preprocess(text))
displacy.render(nlp(doc.text),style='ent',jupyter=True)

It gives me something like:

How can I tag my data because without proper tagging of dates, it's all futile as it'll never learn to get the dates no matter what. Is there any Regular expression RE or I saw that NLTK POS based Queries to extract NER gives us something like:

Answer 1

If I understand your problem correctly, you need to robustly parse dates which are given mostly in a fully textual form.

To normalize such dates you can attempt to use one of dateutil , maya , pendulum or arrow libraries. Good demonstration of possibilities of these libraries you can find here .

In case your problem is also to tag the dates in text, it is a bit more nuanced and you would have to train such a model if SpaCy / NLTK is not suitable for your purpose. You can theoretically also implement a number of regular expressions but this is error prone and would take months to complete. AFAIK, there is no robust implementation available.