使用 Scrapy 和 Python 2.7 抓取网页

Question

Link: http://content.time.com/time/covers/0,16641,19230303,00.html [new DOM link]

Cover Page Html tag

How to get that SCR in Jason and download images

Next Button Tag

I want to scrap this 2 links using Scrapy

Any Help !!

I need to write a method to download images and click on next page, run them in for loop till final image get the download(Final Page). how to download rest of part ill figure it out.

I follow this tutorial https://www.pyimagesearch.com/2015/10/12/scraping-images-with-python-and-scrapy/

[DOM is already outdated ]

I've already set all files and Pipelines for project

For Record, I tried different Different method XPath css response

Answer 1

https://github.com/Dhawal1306/Scrapy

Everything is done solution is on Github 4700 somewhere images we have and along with JSON also.

for a tutorial, any question you just have to ask !!

Answer 2

I know this is not scrapy but I found easier using BS4. so you have to "pip install beautifulsoup4". Here is a sample :

import requests
from bs4 import BeautifulSoup
import os

r = requests.get("https://mouradcloud.westeurope.cloudapp.azure.com/blog/blog/category/food/")
data = r.text
soup = BeautifulSoup(data, "lxml")

for link in soup.find_all('img'):
        image_url = link.get("src")
        print(image_url)

It worked like a charm