提取BeautifulSoup中div的属性值

Question

I am trying to scrape the, 'project' value (the url)from the following. Is it possible to do? Any guidance will be highly appreciated.

<div class="js-project-group">
<div class="grid-row flex flex-wrap">
<div class="js-react-proj-card grid-col-12 grid-col-6-sm grid-col-4-lg" data-pid="564032676" data-project='{
           "id":564032676,
           "name":"SONOFF NSPanel Smart Scene Wall Switch",
           "urls":{"web":{"project":"https://www.kickstarter.com/projects/sonoffnspanel/sonoff-nspanel-smart-scene-wall-switch"}}}'>

           SOME CONTENT 
</div>
<div>SAME ALIKE DIV AS ABOVE</div>
</div> #closing div of grid-row flex flex-wrap
<div>SAME ALIKE DIV AS THE FIRST</div>
</div>

EDIT My try:

projectlist = soup.find_all('div', class_='js-project-group')
projectdata =[]
for project in projectlist:
    tag = project.find('div', class_="js-react-proj-card grid-col-12 grid-col-6-sm grid-col-4-lg")
    attribute = tag['data-project']
    projectdata.append(attribute)

It just gets only the project data in the first div. What I have done wrong?

EDIT Entire HTML structure:

<div class="js-project-group" >
  <div class="grid-row flex flex-wrap">
    <div class="js-react-proj-card></div>
    <div class="js-react-proj-card></div>
    <div class="js-react-proj-card></div>
  </div>
  <div class="grid-row flex flex-wrap">
    <div class="js-react-proj-card></div>
    <div class="js-react-proj-card></div>
    <div class="js-react-proj-card></div>
  </div>
</div>

Answer 1

You should definitely invest a little more time in your questioning, so that the scenario is comprehensible and reproducible for everyone - next time you will do it better - right? -> Check this documentation of minimal reproducible example

What happens?

You try to scrape all projects but still selected the container. That means for your loop, that it iterates only one time.

How to fix?

Select all the projects in the container and iterate:

soup.select('div.js-project-group div.js-react-proj-card.grid-col-12.grid-col-6-sm.grid-col-4-lg')

To get the url you can json.loads() your attribut and navigate to the project url:

json.loads(attribute)['urls']['web']['project']

Example (based on category technology / electronic )

import requests
import json

url = "https://www.kickstarter.com/discover/categories/technology/diy%20electronics"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}

response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.content,'lxml')



projectdata =[]
for project in soup.select('div.js-project-group div.js-react-proj-card.grid-col-12.grid-col-6-sm.grid-col-4-lg'):
    attribute = project['data-project']
    projectdata.append(json.loads(attribute)['urls']['web']['project'])
projectdata

Output

['https://www.kickstarter.com/projects/albertgajsak/circuitmess-batmobile',
 'https://www.kickstarter.com/projects/udoo/udoo-key-the-4-ai-platform',
 'https://www.kickstarter.com/projects/sonoffnspanel/sonoff-nspanel-smart-scene-wall-switch',
 'https://www.kickstarter.com/projects/olman/smart-trays',
 'https://www.kickstarter.com/projects/sbcshop1/pico-led-cube-on-raspberry-pi-pico',
 'https://www.kickstarter.com/projects/fluffee-anion-brush/fluffee-anion-pet-grooming-brush',
 'https://www.kickstarter.com/projects/172204344/ninja-counter-an-arduino-and-raspberry-pi-programmable-timer',
 'https://www.kickstarter.com/projects/amritsingh/pico-air-monitoring-expansion-measure-pm-level-with-pico',
 'https://www.kickstarter.com/projects/eddietay/retro-dreamer-g4a-cm4-by-my-retro-game-case',
 'https://www.kickstarter.com/projects/1405172335/self-powered-rotating-pumpkin-pedestal',
 'https://www.kickstarter.com/projects/udoo/udoo-android-linux-arduino-in-a-tiny-single-board',
 'https://www.kickstarter.com/projects/udoo/udoo-neo-raspberry-pi-arduino-wi-fi-bt-40-sensors']