How do you process invalid HTML in PHP?

Question

I've seen this question , which is very nice and informative. However, it doesn't deal with a rather common scenario.

Say I need to scrape a multitude of websites (or even pages in the same domain), but the author of that website didn't care enough for his code, and has some seriously malformed code "that kinda works". I need to take information from that website.

How do I do it in this case? Ideally without going í͞ń̡͢͡s̶̢̛á̢̕͘ń̵͢҉e̶̸̢̛.

Is it possible? Do I have to revert to RegExp?

Answer 1

You need a DOM Parser. Php has one . And then there are some alternatives (and more... just google for them). You can even run the "garbled HTML" trhu HTML Purifier if you want.

Answer 2

I don't know how your are scraping the site, but working with RegExp will allow you to add many conditions to the scrap code. This may take time, depending on the number of footprints and your RegExp skills.

You may also use Tidy on the site HTML, but this will lead to strange results as well IMO.

Answer 3

Does it have to be PHP? Python has a wonderful library called Beautiful Soup ( "You didn't write that awful page. You're just trying to get some data out of it" ). From my experience I'd recommend it so much that I'd say if you have the option, write a quick Python script to parse your nodes into a clean file that your PHP can pick up.

(Know PHP is in the title & this doesn't directly answer your question. Apologies if you don't have the option of (or dislike) Python, just wanted to present a good alternative.)