如何使用自然语言处理或其他技术从html提取实体

Question

I am trying to parse entities from web pages that contain a time, a place, and a name. I read a little about natural language processing, and entity extraction, but I am not sure if I am heading down the wrong path, so I am asking here.

I haven't started implementing anything yet, so if certain open source libraries are only suitable for a specific language, that is ok.

A lot of times the data would not be found in sentences, but instead in html structures like lists (eg

• 2013-02-01 - Name of Event - Arena Name

).

The structure of the webpages will be vastly different (some might use lists, some might put them in a table, etc.).

What topics can I research to learn more about how to achieve this? Are there any open source libraries that take into account the structure of html when doing entity extraction? Would extracting these (name, time, place) entities from html be better (or even possible) with machine vision where the CSS styling might make it easier to differentiate important parts (name, time, location) of the unstructured text?

Any guidance on topics/open source projects that I can research would help I think.

Answer 1

Many programming languages have external libraries that generate canonical date-stamps from various formats (eg in Java, using the SimpleDateFormat ). As you say, the structure of the web-pages will be vastly different, but date can be expressed using a small number of variations only, so writing down the regular expressiongs for a few (let's say, half-a-dozen) formats will enable extraction of dates from most, if not all, HTML pages.

Extraction of places and names is harder, however. This is where natural language processing will have to come in. What you are looking for is a Named Entity Recognition system. One of the best open source NER systems is the Standford NER . Before using, you should check out their online demo . The demo has three classifiers (for English) that you can choose from. For most of my tasks, I find their english.all.3class.distsim classifier to be quite accurate.

Note that an NER performs well when the places and names you extract are occurring in sentences. If they are going to occur in HTML labels, this approach is probably not going to be very helpful.