如何使用 Python 从调色板网站 web 刮取调色板？

Question

I'm trying to web scrape a list of color palettes with their rgb values from a color palette design website. The HTML code looks like this for each color palette:

<div class = "item block shadow">    
  <div class="palette">
       <div class="place c4" style="background-color: rgb(34, 14, 36);">...</div>
       <div class="place c3" style="background-color: rgb(52, 32, 86);">...</div>
       <div class="place c2" style="background-color: rgb(84, 84, 197);">...</div>
       <div class="place c1" style="background-color: rgb(99, 156, 217);">...</div>

When I web scraped this information on Python, the output didn't show the "style=..." part:

[<div class="palette">
<div class="place c4"><a href=""></a><span></span></div>
<div class="place c3"><a href=""></a><span></span></div>
<div class="place c2"><a href=""></a><span></span></div>
<div class="place c1"><a href=""></a><span></span></div>
</div>]

Is there a way to extract the information I'm looking for? Thanks in advance.

edit: here's my code

import requests
from bs4 import BeautifulSoup

page = requests.get('https://colorhunt.co/palettes/popular')
soup = BeautifulSoup(page.text, 'html.parser')
repo = soup.find(class_="item block shadow")
repo_list = repo.find_all(class_='palette')

Answer 1

The problem here is not that you scrape the wrong part, it's that you scrape the page before it's done loading.This page is https://colorhunt.co/ btw.

When you run this command page = requests.get('https://colorhunt.co/palettes/popular') . What you're getting is a raw HTML file with no JS execution yet.Take a look at script tag of the html you're getting. You'll see lot of JS code is for rendering elements especially in the palette class sections you're trying to get. Thus, all of the colors are not in your scraped content.

Solution: use a library that simulate as if you're browsing the web from the code, run the webpage, wait for it to finish, when it is done, scrape. I learn JS so my choice is puppeteer , there is also a python equivalent called pyppeteer but I think there are more libraries like these.

Hope this answer will help although it's been over a year since you asked. I ran into this problem and no legitimate answer is found anywhere so this will probably save some others' time. Better be late than nothing:)