stackoom
  简体   繁体   中英

Scraping Produce image by BeautifulSoup

 原文  2022-11-09 18:42:41  3   1    python/ web-scraping/ beautifulsoup

Question

I'm trying to Scrap the product image of this page:

https://www.noon.com/egypt-ar/golden-wood-edp-100ml/N39185122A/p/?o=d55236b5f16d3c9d

but i still have an error, I using beaurifaulSoup, but it give me none:

That what i got:

在此处输入图像描述

import requests
from bs4 import BeautifulSoup
import time
import json
import re



page_numbers = 1




 
while True:

    URL = "https://www.noon.com/egypt-ar/beauty-and-health/beauty/eg-nov22-clearance-2/?limit=50&page="+str(page_numbers)

    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}

    page = requests.get(URL, headers=headers)
    soup = BeautifulSoup(page.text, "html.parser")
    products = soup.find_all("span",class_="sc-5e739f1b-0 gEERDr wrapper productContainer")
    


    time.sleep(1) 
    product_Source =soup.find("div",class_="sc-95314d04-3 dnAXkE")
    product_String = product_Source.text
    all_words = product_String.split()
    products_Numbers= all_words[0]

#loop to Get Data

    for product in products:

        product_link = product.find("a").attrs["href"]                                                         #Get product product links
        
        productFullLink = "https://www.noon.com"+product_link
        Moreinfo = requests.get(productFullLink,headers=headers)                                         # Get informattion from page
        Product_Details = BeautifulSoup(Moreinfo.text,"html.parser")
        #time.sleep(1)

       
            
        product_image = Product_Details.find_all("div",class_="sc-8cbb8e24-2 eDXUdi")
       
        

        print(product_image)

1 answers

solution1
 0  2022-11-10 15:40:45

For me, the following selector works:

product_image = Product_Details.select(".swiper-wrapper .lazyload-wrapper div")
print(product_image)

For the first 3 products, it prints

[<div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر جولدن وود EDP 100مل - v1666705361/N39185122A_1" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1666705361/N39185122A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر جولدن وود EDP 100مل - v1618317832/N39185122A_2" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر جولدن وود EDP 100مل - v1618317832/N39185122A_3" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_3.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر جولدن وود EDP 100مل - v1617954175/N39185122A_5" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_5.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر جولدن وود EDP 100مل - v1617954175/N39185122A_4" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_4.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1666705361/N39185122A_1" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1666705361/N39185122A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1618317832/N39185122A_2" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1618317832/N39185122A_3" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_3.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1617954175/N39185122A_5" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_5.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1617954175/N39185122A_4" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_4.jpg"/></div>]
[<div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر سايلنسز EDP 100مل - v1562242787/N11201003A_1" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر سايلنسز EDP 100مل - v1562243187/N11201003A_2" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1562243187/N11201003A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1562242787/N11201003A_1" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1562242787/N11201003A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1562243187/N11201003A_2" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1562243187/N11201003A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر سايلنسز EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="مزيل عرق فريشلي فري ببكرة دوارة برائحة الجريب فروت والليتشي 50مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1654077085/N42721327A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر سايلنسز EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="مزيل عرق فريشلي فري ببكرة دوارة برائحة الجريب فروت والليتشي 50مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1654077085/N42721327A_1.jpg"/></div>]
[<div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر رويال أمبر EDP 100مل - v1515389873/N11202278A_1" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر رويال أمبر EDP 100مل - v1515389757/N11202278A_2" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1515389757/N11202278A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر رويال أمبر EDP 100مل - v1560784604/N11202278A_3" aria-hidden="true" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1560784604/N11202278A_3.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1515389873/N11202278A_1" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1515389873/N11202278A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1515389757/N11202278A_2" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1515389757/N11202278A_2.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="v1560784604/N11202278A_3" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_400/v1560784604/N11202278A_3.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر رويال أمبر EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر بيور بلوم EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1572605693/N31750079A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر رويال أمبر EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg"/></div>, <div class="sc-8cbb8e24-2 eDXUdi"><img alt="عطر بيور بلوم EDP 100مل" class="sc-b51db3f-1 iCVkuj" src="https://f.nooncdn.com/products/tr:n-t_80/v1572605693/N31750079A_1.jpg"/></div>]

but then, even .find_all("div", class_="sc-8cbb8e24-2 eDXUdi") gives me the same results. Also, wouldn't getting the img tags be more useful? (You can select them with .swiper-wrapper img .)

Are you sure you're fetching the right html? Your headers didn't work for me, and I wasn't able to write a working request at all, so I actually used selenium to get the html. If interested, take a look at my edited version of your code . I replaced your for-loop with reusable functions [after some adjustments] that I use in some of my crawlers, and the output looks like

[
    {
        "page_number": 1,
        "page_link": "https://www.noon.com/egypt-ar/beauty-and-health/beauty/eg-nov22-clearance-2/?limit=50&page=1",
        "products_count": 182,
        "products": [
                {
                "product_id": "N39185122A",
                "product_brand": "إس. تي. ديبون",
                "product_name": "عطر جولدن وود EDP 100مل",
                "product_price_text": "رقم الموديل : 16904A قبل : 1298.00 جنيه الآن : 975.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 323.00 جنيه خصم 24٪",
                "current_price": "975.00",
                "product_link": "https://www.noon.com/egypt-ar/golden-wood-edp-100ml/N39185122A/p/?o=d55236b5f16d3c9d",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1666705361/N39185122A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_5.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_4.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1666705361/N39185122A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_5.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_4.jpg"]
            },
            {
                "product_id": "N11201003A",
                "product_brand": "جاكومو باريس",
                "product_name": "عطر سايلنسز EDP 100مل",
                "product_price_text": "4.3 12 تقييمات قبل : 811.00 جنيه الآن : 675.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 136.00 جنيه خصم 16٪",
                "current_price": "675.00",
                "product_link": "https://www.noon.com/egypt-ar/silences-edp-100ml/N11201003A/p/?o=b886ef68a4cc3e4c",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1562243187/N11201003A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1562242787/N11201003A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1562243187/N11201003A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1654077085/N42721327A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1562242787/N11201003A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1654077085/N42721327A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg"]
            }
        ]
    },
    {
        "page_number": 2,
        "page_link": "https://www.noon.com/egypt-ar/beauty-and-health/beauty/eg-nov22-clearance-2/?limit=50&page=2",
        "products_count": 182,
        "products": [
                {
                "product_id": "N42677090A",
                "product_brand": "جاكومو",
                "product_name": "طَقم هدايا مكون من أو دي بارفان وبخاخ مناسب للسفر من قطعتين 100مل",
                "product_price_text": "رقم الموديل : 3392865093684 قبل : 1241.00 جنيه الآن : 679.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 562.00 جنيه خصم 45٪",
                "current_price": "679.00",
                "product_link": "https://www.noon.com/egypt-ar/2-piece-eau-de-parfum-and-travel-spray-gift-set-100ml/N42677090A/p/?o=d8dc699010d8086b",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1608984530/N42677090A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1608984530/N42677090A_1.jpg"]
            },
            {
                "product_id": "N11200698A",
                "product_brand": "لانفن باريس",
                "product_name": "ماء تواليت إكلا داربيج للرجال 100مل",
                "product_price_text": "5.0 2 تقييمات قبل : 1365.00 جنيه الآن : 687.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 678.00 جنيه خصم 49٪",
                "current_price": "687.00",
                "product_link": "https://www.noon.com/egypt-ar/eclat-d-arpege-pour-homme-edt-100ml/N11200698A/p/?o=c5824e03ee50f26a",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1544679664/N11200698A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1544679664/N11200698A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1544679665/N11200698A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1544679664/N11200698A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1544679664/N11200698A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1544679665/N11200698A_3.jpg"]
            }
        ]
    },
    {
        "page_number": 3,
        "page_link": "https://www.noon.com/egypt-ar/beauty-and-health/beauty/eg-nov22-clearance-2/?limit=50&page=3",
        "products_count": 182,
        "products": [
                {
                "product_id": "N39185122A",
                "product_brand": "إس. تي. ديبون",
                "product_name": "عطر جولدن وود EDP 100مل",
                "product_price_text": "رقم الموديل : 16904A قبل : 1298.00 جنيه الآن : 975.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 323.00 جنيه خصم 24٪",
                "current_price": "975.00",
                "product_link": "https://www.noon.com/egypt-ar/golden-wood-edp-100ml/N39185122A/p/?o=d55236b5f16d3c9d",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1666705361/N39185122A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1618317832/N39185122A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_5.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1617954175/N39185122A_4.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1666705361/N39185122A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1618317832/N39185122A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_5.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1617954175/N39185122A_4.jpg"]
            },
            {
                "product_id": "N11202278A",
                "product_brand": "إس. تي. ديبون",
                "product_name": "عطر رويال أمبر EDP 100مل",
                "product_price_text": "4.7 9 تقييمات قبل : 1899.00 جنيه الآن : 699.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 1200.00 جنيه خصم 63٪",
                "current_price": "699.00",
                "product_link": "https://www.noon.com/egypt-ar/royal-amber-edp-100ml/N11202278A/p/?o=dd94cf4150e7d47c",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1515389757/N11202278A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1560784604/N11202278A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1515389873/N11202278A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1515389757/N11202278A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1560784604/N11202278A_3.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1572605693/N31750079A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1515389873/N11202278A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg", "https://f.nooncdn.com/products/tr:n-t_80/v1572605693/N31750079A_1.jpg", "https://z.nooncdn.com/s/app/com/noon/icons/checkbox-square_checked_v2.svg", "https://z.nooncdn.com/s/app/com/noon/images/fulfilment_express_v2-ar.svg"]
            }
        ]
    },
    {
        "page_number": 4,
        "page_link": "https://www.noon.com/egypt-ar/beauty-and-health/beauty/eg-nov22-clearance-2/?limit=50&page=4",
        "products_count": 182,
        "products": [
                {
                "product_id": "N31126667A",
                "product_brand": "فيكتور أند رولف",
                "product_name": "ماء عطر فلاور بومب ميدنايبت 100مل",
                "product_price_text": "رقم الموديل : 10507A 4.5 2 تقييمات قبل : 2999.00 جنيه الآن : 2395.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 604.00 جنيه خصم 20٪",
                "current_price": "2395.00",
                "product_link": "https://www.noon.com/egypt-ar/flowerbomb-midnight-edp-100ml/N31126667A/p/?o=e34956fccd70220b",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1572005046/N31126667A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_80/v1572005047/N31126667A_2.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1572005046/N31126667A_1.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1572005047/N31126667A_2.jpg"]
            },
            {
                "product_id": "N42264463A",
                "product_brand": "كييلز",
                "product_name": "محلول تصحيح لعلاج البقع الداكنة بوضوح 30مل",
                "product_price_text": "رقم الموديل : 3605970202637 5.0 تقييم عميل واحد قبل : 2678.57 جنيه الآن : 1715.00 جنيه يشمل ضريبة القيمة المضافة وفّرت : 963.57 جنيه خصم 35٪",
                "current_price": "1715.00",
                "product_link": "https://www.noon.com/egypt-ar/clearly-corrective-dark-spot-solution-30ml/N42264463A/p/?o=bba8c04cdb7cce0e",
                "imageLinks_list": ["https://f.nooncdn.com/products/tr:n-t_80/v1605337673/N42264463A_5.jpg", "https://f.nooncdn.com/products/tr:n-t_400/v1605337673/N42264463A_5.jpg"]
            }
        ]
    }
]

(only 2 products from each page included)

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Error while scraping image with beautifulsoup Scraping Google image search Python (requests, beautifulsoup) Python / BeautifulSoup Image Scraping Does Not Save Animated GIFs Correctly Export BeautifulSoup scraping results to CSV; scrape + include image values in column Scraping image hrefs from an Ordered List using BeautifulSoup Scraping Craiglist with BeautifulSoup and getting first image in each posting BeautifulSoup, Scraping, get image size without being in the tags? Web scraping both html text and image link with Python Beautifulsoup Web scraping (Graph Scraping) with BeautifulSoup scraping tables with beautifulsoup
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM