Website error after scraping

Question

I made a simple scraper that accesses an album, and scrapes lyrics for each song from azlyrics.com.

After about an hour of working, the website crashed, with an error:

Chrome:

www.azlyrics.com didn't send any data. ERR_EMPTY_RESPONSE

Tor, firefox, waterfox:

The connection was reset The connection to the server was reset while the page was loading.

It's the same for all devices on my home network. If I use mobile data to access it via my phone it works fine.

I tried fixing it with ipconfig /release /renew, but it didn't work. I'm at a loss for what else I could do or why it even happened. Any help is greatly appreciated.

Answer 1

Apparently your IP was banned by the website for suspicious activity. There are couple ways around that:

• talk to website owners. This is the most straightforward and nicest way
• change your IP, eg by connecting though a pool of public proxies or Tor. This is a little bit dirty and it is not so robust, eg you can be banned by user-agent or some other properties of your scraper.