Scraping Information from multiple URLS that are different in structure

Question

I would like to scrape multiple URLS but they are of different nature, such as different company websites with different html backend. Is there a way to do it without coming up with a customised code for each url?

Understand that I can put multiple URLS into a list and loop them

Answer 1

I fear not, but I am not an expert:-)

I could imagine that it depends on the complexity of the structures. If you want to find a the text "Test" on every website, I coul imagine that soup.body.findAll(text='Test') would return all occurences of "Test" on the website.

I assume you're aware of how to loop through a list here, so that you'd loop through the list of URLS and for each check whether the searched string occurs (maybe you are looking for sth else, ie an "apply" button or "login"?

all the best,