How Can I Locate Disabled With Beautiful Soup?

Question

I cannot find disabled on webpage: https://sport.woot.com/offers/asics-mens-clearance-calculator?ref=w_cnt_lnd_cat_sprt_18_1

<div class="attribute">
    <label for="attr-size">Size</label>

    <select id="attr-size" name="Size">
        <option value="none">- Select -</option>
        <option value="6.5">6.5</option>
        <option value="8">8</option>
        <option value="8.5" disabled="">8.5</option>
        <option value="9.5">9.5</option>
        <option value="10">10</option>
        <option value="10.5">10.5</option>
        <option value="11">11</option>
        <option value="12" disabled="">12</option>
        <option value="13" disabled="">13</option>
        <option value="14">14</option>
        <option value="12 Wide US">12 Wide US</option>
        <option value="13 (4E)" disabled="">13 (4E)</option>
    </select>
</div>

I've tried:

import requests
from bs4 import BeautifulSoup

result = requests.get("https://sport.woot.com/offers/asics-mens-clearance- 
calculator?ref=w_cnt_lnd_cat_sprt_18_1")
src = result.content
soup = BeautifulSoup(src, 'lxml')
print(soup.has_attr('disabled'))

and

import requests
from bs4 import BeautifulSoup

result = requests.get("https://sport.woot.com/offers/asics-mens-clearance- 
calculator?ref=w_cnt_lnd_cat_sprt_18_1")
src = result.content
soup = BeautifulSoup(src, 'lxml')
print(soup.get('option', {'value': 'disabled'}))

I'm parsing information from woot.com

Answer 1

This page uses JavaScript to set disabled elements but requests / BeautifulSoup can't run JavaScript . You may need Selenium to control web browser which can run JavaScript .

OR you can try to get url which JavaScript ( AJAX / XHR ) uses to get data from server.

But this page doesn't use AJAX/XHR to load information. In HTML I see script with

var offerItems = [...] 

which probably have all information

---

This script get all 6th <script> and cut off 4th line which have var offerItems = (I found this position checking code manually)

Next it cuts off only text with data in [ ] and use module json to convert it to Python' list/dictionary.

Later it can easily get values from this list/dictionary.

It has only information Size and Quantity so you would have to add code which calculates which sizes are missing so they can be disabled. I skiped this part - I didn't even try to calculate it.

BTW: Some code I wrote in many lines to make it more readable but you can write it in one line.

import requests
from bs4 import BeautifulSoup
import json

response = requests.get("https://sport.woot.com/offers/asics-mens-clearance-calculator?ref=w_cnt_lnd_cat_sprt_18_1")
soup = BeautifulSoup(response.content, 'lxml')

all_scripts = soup.find_all('script')
#for i, item in enumerate(all_scripts):
#    print('---', i, '---')
#    print(item.text[:200])

text = all_scripts[5].text
lines = text.split('\n')
line = lines[3].strip()
line = line[len("var offerItems = "):-1]

data = json.loads(line)

#sizes = ["6.5", "8", "8.5", "9.5", "10", "10.5", "11", "12", "13", "14", "12 Wide US", "13 (4E)"]

print(data[0].keys())
for item in data:
    print('------------------------------------')
    print('Key:', item['Key'])
    print('Quantity:', item['Quantity'])
    for x in item['Attributes']:
        print('> ', x['Key'], ':', x['Value'])
    #for key, value in item.items():
    #    print('>', key, ':', value)

Result:

dict_keys(['Id', 'Asin', 'AmazonPrice', 'Attributes', 'FormattedDiscount', 'FormattedListPrice', 'FormattedSalePrice', 'FormattedWarrantyPrice', 'Key', 'ListPrice', 'Quantity', 'SalePrice', 'SKU', 'TotalReviews', 'WarrantyPrice', 'WarrantyTerm', 'WarrantyUrl'])

------------------------------------
Key: 6.5|Gel-Nimbus 19 - Black/Onyx/Silver
Quantity: 10
>  Size : 6.5
>  Model : Gel-Nimbus 19 - Black/Onyx/Silver
------------------------------------
Key: 8|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 6
>  Size : 8
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 8|GT-2000 7 - Illusion Blue/Black
Quantity: 5
>  Size : 8
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 8|GT-2000 6 - Everglade/ Black
Quantity: 10
>  Size : 8
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 8.5|Frequent Trail - Black/ Birch
Quantity: 0
>  Size : 8.5
>  Model : Frequent Trail - Black/ Birch
------------------------------------
Key: 8.5|GT-2000 7 - Illusion Blue/Black
Quantity: 0
>  Size : 8.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Venture 6 - Frost Grey/ Phantom/ Black
Quantity: 0
>  Size : 9.5
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Illusion Blue/Black
Quantity: 3
>  Size : 9.5
>  Model : GT-2000 7 - Illusion Blue/Black
------------------------------------
Key: 9.5|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 10
>  Size : 9.5
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 10|GT-2000 6 - Everglade/ Black
Quantity: 4
>  Size : 10
>  Model : GT-2000 6 - Everglade/ Black
------------------------------------
Key: 9.5|GT-2000 7 - Black/White
Quantity: 0
>  Size : 9.5
>  Model : GT-2000 7 - Black/White
------------------------------------
Key: 9.5|GT-2000 6 - Dark Grey/Black/Fiery Red
Quantity: 9
>  Size : 9.5
>  Model : GT-2000 6 - Dark Grey/Black/Fiery Red
------------------------------------
Key: 10|GT-2000 8 - Piedmont Grey/Black
Quantity: 3
>  Size : 10
>  Model : GT-2000 8 - Piedmont Grey/Black
------------------------------------
Key: 10|Gel-Contend 4 - Carbon/ Classic Red/ Black
Quantity: 6
>  Size : 10
>  Model : Gel-Contend 4 - Carbon/ Classic Red/ Black
------------------------------------
Key: 10.5|Gel-Excite 4 - Silver/Black/Imperial
Quantity: 10
>  Size : 10.5
>  Model : Gel-Excite 4 - Silver/Black/Imperial
------------------------------------
Key: 10.5|Gel-Excite 4 - Carbon/ Silver/ Black
Quantity: 0
>  Size : 10.5
>  Model : Gel-Excite 4 - Carbon/ Silver/ Black
------------------------------------
Key: 10.5|Gel-Nimbus 21 - Black/Classic Red
Quantity: 1
>  Size : 10.5
>  Model : Gel-Nimbus 21 - Black/Classic Red
------------------------------------
Key: 11|Gel-Nimbus 20 - Island Blue/White/Black
Quantity: 5
>  Size : 11
>  Model : Gel-Nimbus 20 - Island Blue/White/Black
------------------------------------
Key: 12|Gel-Nimbus 19 - Glacier Grey/Silver/White
Quantity: 0
>  Size : 12
>  Model : Gel-Nimbus 19 - Glacier Grey/Silver/White
------------------------------------
Key: 13|GT-1000 7 - Grand Shark/ Peacoat
Quantity: 0
>  Size : 13
>  Model : GT-1000 7 - Grand Shark/ Peacoat
------------------------------------
Key: 14|Gel-Cumulus 20 - Race Blue/ Peacoat
Quantity: 1
>  Size : 14
>  Model : Gel-Cumulus 20 - Race Blue/ Peacoat
------------------------------------
Key: 14|Gel-Nimbus 20 - Indigo blue/fiery red
Quantity: 0
>  Size : 14
>  Model : Gel-Nimbus 20 - Indigo blue/fiery red
------------------------------------
Key: 12 Wide US|Gel-Nimbus 20 - Black/White/Carbon
Quantity: 3
>  Size : 12 Wide US
>  Model : Gel-Nimbus 20 - Black/White/Carbon
------------------------------------
Key: Gel-Venture 6 - Frost Grey/ Phantom/ Black|13 (4E)
Quantity: 0
>  Model : Gel-Venture 6 - Frost Grey/ Phantom/ Black
>  Size : 13 (4E)