从第三方网站获取数据

Question

I work in a small healthcare related office and we often have to look up license and other related official numbers of physicians. We use websites that are free and available to the public to do so. I've been tasked with figuring out a way to enter in the physician name and then return the results from all of the websites in a single entry to reduce the amount of time spent going through each website. I'm familiar with javascript, php and ruby but by no means an expert. My question is, where should I start? I don't need anyone to write the code for me or anything, but I can't seem to form the right question to google for some answers. I'm fairly sure this is possible, just not sure where to start developing my idea. Any help would be appreciated.

Answer 1

It sounds like you need to do some screen scraping, which may or may not be allowed by the terms and conditions of the sites you're using - you should check that first.

If there aren't any restrictions on automatic retrieval and querying, you'll want to read up on PHP's cURL module, and simulate the form actions that are performed when you manually query the sites. You can use your browser's developer console to see what scripts and pages are called when you run queries - it's quicker than trying to work it out from the page source.

You'll get back the HTML from the pages, which you'll need to parse. Depending on the format on the page, a few simple regexes might do the trick, but you'll likely need to tailor them for each site you query.

Again, please double check that the sites you're using allow you to run scripted queries - if you're in any doubt, you should email them and explain what you plan to do, and ask if they're ok with it.