从用于 NLP 的非结构化医学文档中提取文本

Question

I have a lot of unstructured medical documents in all sorts of different formats. What's the best way to parse out all the good sentences to use for NLP?

Currently I'm using SpaCy to do this, but even with multiprocessing it is pretty slow, and and the default sentence parser doesn't work 100% of the time. Here is an example of how I try and get good sentences with SpaCy:

def get_good_sents(texts, batch_size, n_process):
    nlp = spacy.load("en_core_web_sm", disable=[
        'ner',
        'entity_linker',
        'textcat',
        'entity_ruler',
        'sentencizer',
        'merge_noun_chunks',
        'merge_entities',
        'merge_subtokens',
    ])
    pipe = nlp.pipe(texts, batch_size=batch_size, n_process=n_process)

    rows = []
    for doc in pipe:
        clean_text = []
        for sent in doc.sents:
            struct = [token.pos_ for token in sent]
            subject = any(x in struct for x in ['NOUN', 'PRON'])
            action = any(x in struct for x in ['VERB', 'ADJ', 'AUX'])

            if subject and action :
                clean_text.append(sent.text)
        rows.append(' '.join(clean_text).replace('\n', ' ').replace('\r', ''))

    return rows

Example of some text extracts

Raw Text:

TITLE
Patient Name:
Has a heart Condition.
Is 70 Years old.

Expected Output:

Has a heart Condition.
Is 70 Years old.

This examples not great because I have tons of different documents in all sort of various formats. They can really vary a lot. It basically boils down to me just wanting to strip out the boiler plate stuff and just get the actual free text.

Answer 1

Based on the comments from the above discussion, I am very confident that spaCy will not provide you with very good results, simply because it is very much tied to the expectation of a valid grammatical sentence.

At least with the current approach of looking for "correctly tagged words" in each line, I would expect this to not work very well, since tagging a sentence correctly is already tied to a decent input format; it is once again time to quote one of my favorite concepts in Machine Learning .

Depending on the accuracy you want to achieve, I would personally adopt a defensive Regex approach, where you manually sort out headings (lines with fewer than 4 words, lines that end in a colon/semicolon, etc.), although it will require significantly more effort.

Another, more direct solution would be to take what other common boilerplate tools are doing , although most of those are targeted to remove boilerplate from HTML content, and thus have an easier time by utilizing tag information as well.