assistance with scraping json parse using bs4
Question
So I successfully scraped the info of the json parse on this website, but I want to extract "all" of a specific data from the 'variants' array for example the 'inventory_quantity'. When I run my code it only shows the 1st array of the 'inventory_quantity' but I want it to extract all of the 'inventoryquantity' instead of just the 1st one.
import requests
import json
from bs4 import BeautifulSoup
url = 'https://www.amongstfew.com/drops'
def scrape():
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
script = soup.find("script", class="draw_data_json").string.strip()
data = json.loads(script)
for i in data:
print(i['title'], i['variants'][0]['inventory_quantity'])
scrape()
2 answers
solution1
0 2021-05-07 10:05:06
Your data is list of dictionary so you can loop over it using range and len in python and also other inventory_quantity is inside variants you can loop over it and extract the information
code:
import requests
import json
from bs4 import BeautifulSoup
url = 'https://www.amongstfew.com/drops'
def scrape():
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
script = soup.find("script", class="draw_data_json").string.strip()
data = json.loads(script)
for i in range(len(data)):
for j in range(len(data[i]['variants'])):
print(data[i]['title'], data[i]['variants'][j]['inventory_quantity'])
scrape()
output:
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 2
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 2
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM) 1
....
solution2
0 2021-05-07 13:31:23
The variants are just in a list. Iterate through those. Note when you iterate through a list there is no need for doing range(1,len(data)) . You can simply slice the list with list[1,:]
import requests
import json
from bs4 import BeautifulSoup
url = 'https://www.amongstfew.com/drops'
def scrape():
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
script = soup.find("script", {'class':"draw_data_json"}).string.strip()
data = json.loads(script)
for i in data:
print('%s\n%s\n%s' %('*'*60,i['title'].center(60, " "),'*'*60))
for variant in i['variants']:
print('\t %-30s %s' %(variant['title'], variant['inventory_quantity']))
print('\n\n')
scrape()
Output:
************************************************************
adidas - Yeezy 500 (ENFLAM/ENFLAM/ENFLAM)
************************************************************
US 5.5 / UK 5 / EU 38 1
US 6 / UK 5.5 / EU 38 2/3 1
US 6.5 / UK 6 / EU 39 1/3 1
US 7 / UK 6.5 / EU 40 1
US 7.5 / UK 7 / EU 40 2/3 1
US 8 / UK 7.5 / EU 41 1/3 1
US 8.5 / UK 8 / EU 42 1
US 9 / UK 8.5 / EU 42 2/3 2
US 9.5 / UK 9 / EU 43 1/3 2
US 10 / UK 9.5 / EU 44 1
US 10.5 / UK 10 / EU 44 2/3 1
US 11 / UK 10.5 / EU 45 1/3 1
US 11.5 / UK 11 / EU 46 1
************************************************************
adidas - Yeezy BOOST 350 V2 (Sesame/Sesame/Sesame)
************************************************************
US 5 / UK 4.5 / EU 37 1/3 0
US 5.5 / UK 5 / EU 38 0
US 6 / UK 5.5 / EU 38 2/3 0
US 6.5 / UK 6 / EU 39 1/3 0
US 7 / UK 6.5 / EU 40 0
US 7.5 / UK 7 / EU 40 2/3 0
US 8 / UK 7.5 / EU 41 1/3 0
US 8.5 / UK 8 / EU 42 0
US 9 / UK 8.5 / EU 42 2/3 0
US 9.5 / UK 9 / EU 43 1/3 0
US 10 / UK 9.5 / EU 44 0
US 10.5 / UK 10 / EU 44 2/3 0
US 11 / UK 10.5 / EU 45 1/3 0
US 11.5 / UK 11 / EU 46 0
US 12 / UK 11.5 / EU 46 2/3 0
US 12.5 / UK 12 / EU 47 1/3 0
************************************************************
Air Jordan 1 High OG WMNS (Lucky Green/Black-White-Sail)
************************************************************
US 5 / UK 2.5 / EU 35.5 0
US 5.5 / UK 3 / EU 36 0
US 6 / UK 3.5 / EU 36.5 0
US 6.5 / UK 4 / EU 37.5 0
US 7 / UK 4.5 / EU 38 0
US 7.5 / UK 5 / EU 38.5 0
US 8 / UK 5.5 / EU 39 0
US 8.5 / UK 6 / EU 40 0
US 9 / UK 6.5 / EU 40.5 0
US 9.5 / UK 7 / EU 41 0
US 10 / UK 7.5 / EU 42 0
US 10.5 / UK 8 / EU 42.5 0
US 11 / UK 8.5 / EU 43 0
US 11.5 / UK 9 / EU 44 0
US 12 / UK 9.5 / EU 44.5 0
************************************************************
Air Jordan 1 Low (Sail/White-LT Zitron)
************************************************************
US 7 / UK 6 / EU 40 0
US 7.5 / UK 6.5 / EU 40.5 0
US 8 / UK 7 / EU 41 0
US 8.5 / UK 7.5 / EU 42 0
US 9 / UK 8 / EU 42.5 1
US 9.5 / UK 8.5 / EU 43 2
US 10 / UK 9 / EU 44 0
US 10.5 / UK 9.5 / EU 44.5 1
US 11 / UK 10 / EU 45 1
US 11.5 / UK 10.5 / EU 45.5 1
US 12 / UK 11 / EU 46 0
US 13 / UK 12 / EU 47.5 1
************************************************************
Air Jordan 1 Zoom Air CMFT (Sail/White-LT Zitron)
************************************************************
US 7 / UK 6 / EU 40 1
US 7.5 / UK 6.5 / EU 40.5 0
US 8 / UK 7 / EU 41 1
US 8.5 / UK 7.5 / EU 42 0
US 9 / UK 8 / EU 42.5 0
US 9.5 / UK 8.5 / EU 43 0
US 10 / UK 9 / EU 44 0
US 10.5 / UK 9.5 / EU 44.5 0
US 11 / UK 10 / EU 45 0
US 11.5 / UK 10.5 / EU 45.5 0
US 12 / UK 11 / EU 46 1
US 12.5 / UK 11.5 / EU 47 0
US 13 / UK 12 / EU 47.5 1
************************************************************
Air Jordan 11 Retro Low (White/Legend Blue-White-Black)
************************************************************
US 7 / UK 6 / EU 40 0
US 7.5 / UK 6.5 / EU 40.5 1
US 8 / UK 7 / EU 41 0
US 8.5 / UK 7.5 / EU 42 0
US 9 / UK 8 / EU 42.5 0
US 9.5 / UK 8.5 / EU 43 0
US 10 / UK 9 / EU 44 0
US 10.5 / UK 9.5 / EU 44.5 0
US 11 / UK 10 / EU 45 1
US 11.5 / UK 10.5 / EU 45.5 0
US 12 / UK 11 / EU 46 0
US 12.5 / UK 11.5 / EU 47 0
US 13 / UK 12 / EU 47.5 0
US 14 / UK 13 / EU 48.5 0
************************************************************
Air Jordan 4 X Union Retro SP (Off Noir/Red/Blue)
************************************************************
US 4 / UK 3.5 / EU 36 0
US 4.5 / UK 4 / EU 36.5 0
US 5 / UK 4.5 / EU 37.5 0
US 5.5 / UK 5 / EU 38 0
US 6 / UK 5.5 / EU 38.5 0
US 6.5 / UK 6 / EU 39 0
US 7 / UK 6 / EU 40 0
US 7.5 / UK 6.5 / EU 40.5 0
US 8 / UK 7 / EU 41 0
US 8.5 / UK 7.5 / EU 42 0
US 9 / UK 8 / EU 42.5 0
US 9.5 / UK 8.5 / EU 43 0
US 10 / UK 9 / EU 44 0
US 10.5 / UK 9.5 / EU 44.5 0
US 11 / UK 10 / EU 45 0
US 11.5 / UK 10.5 / EU 45.5 0
US 12 / UK 11 / EU 46 0
US 12.5 / UK 11.5 / EU 47 0
US 13 / UK 12 / EU 47.5 0
US 14 / UK 13 / EU 48.5 0
US 15 / UK 14 / EU 49.5 0
************************************************************
Nike - Air Max 1 PRM (Pine Green/Pine Green)
************************************************************
US 5 / UK 4.5 / EU 37.5 0
US 5.5 / UK 5 / EU 38 0
US 6 / UK 5.5 / EU 38.5 0
US 6.5 / UK 6 / EU 39 0
US 7 / UK 6 / EU 40 0
US 7.5 / UK 6.5 / EU 40.5 0
US 8 / UK 7 / EU 41 0
US 8.5 / UK 7.5 / EU 42 0
US 9 / UK 8 / EU 42.5 0
US 9.5 / UK 8.5 / EU 43 0
US 10 / UK 9 / EU 44 0
US 10.5 / UK 9.5 / EU 44.5 0
US 11 / UK 10 / EU 45 0
US 11.5 / UK 10.5 / EU 45.5 0
US 12 / UK 11 / EU 46 0
************************************************************
Nike - WMNS Dunk Low (Lemon Drop/Opti Yellow-Sail)
************************************************************
US 5.5 / UK 3 / EU 36 1
US 6 / UK 3.5 / EU 36.5 1
US 6.5 / UK 4 / EU 37.5 1
US 7 / UK 4.5 / EU 38 1
US 7.5 / UK 5 / EU 38.5 1
US 8 / UK 5.5 / EU 39 1
US 8.5 / UK 6 / EU 40 1
US 9 / UK 6.5 / EU 40.5 1
US 9.5 / UK 7 / EU 41 1
US 10 / UK 7.5 / EU 42 1
US 10.5 / UK 8 / EU 42.5 1
US 11 / UK 8.5 / EU 43 1
US 11.5 / UK 9 / EU 44 1
US 12 / UK 9.5 / EU 44.5 1
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Scraping PubMed using bs4
scraping using BS4 python
Scraping using SELENIUM/BS4
Scraping using BS4 with time
Scraping Script tag in HTML with Json and BS4
Using requests with bs4 and or json
Scraping with BS4 - using regex - storing in sqlite
Scraping links using bs4 and python
Web scraping using selenium and bs4
Web Scraping using bs4 with Python
Related Tags