python BeautifulSoup not getting text from webpage

Question

I am trying to get product name from the webpage using python. but it returns only a empty tags. i also tried requests library and lxml parsing in BeautifulSoup . pls help me to fix this problem, thanks in advance :-)

HTML in site:

<div class="product-name">SWAN</div>
   <div class="product-price">
   <span class="final-price">₹10650</span>
</div>
<div class="specification">
   <div>Specifications</div>
   <table>
      <tr>
         <td>....</td>
      </tr>
      <tr>
         <td>....</td>
     </tr>
   </table>
</div>

python code:

url = "http://opor.in/ProductDetail/Index?ProductId=212"
page = urlopen(url).read()
html = bs(page, 'html.parser')
model_name = html.find('div', attrs={'class':'product-name'})
spec = html.find('div', attrs={'class':'specification'})
print(model_name)
print(spec)

Output:

<div class="product-name"></div>
<div class="specification">
<div>Specifications</div>
<table></table>
</div>

Answer 1

The data loaded by java-scripts.However if you see the DOM data available in script tag.To fetch the value from script tag and load into json and then get the key value.

Code :

from urllib.request import urlopen
from bs4 import BeautifulSoup as bs
import json
url = "http://opor.in/ProductDetail/Index?ProductId=212"
page = urlopen(url).read()
soup = bs(page, 'html.parser')

for item in soup.find_all('script'):
   if 'productDetail' in item.text:
       data=item.text.split('var productDetail =')[-1].split('};')[0] + "}"
       datajson=json.loads(data.strip())
       print('Product Code :' + datajson['ProductCode'])
       for item in datajson['ProductSpecification']:
           print(item['SpecificationName'] + " : "+ item['SpecificationValue'])

Output :

Product Code :1601KFMB
MEMBRANE : MEMBRELLA -ALPHA- 80 GPD (2 NOS)
PUMP : KEMFLO 48 V
APPLICATION : SUITABLE FOR BRACKISH WATER
FILTER LIFE : APPROX 3000 LITRE / 6 MONTHS
FILTERS : SEDIMENT, PRECARBON, POST CARBON
FLOAT : MEMBRELLA
FR : MEMBRELLA /KFL
INLINE SET : MEMBRELLA
INPUT VOLTAGE : 100-300 VOLT AC (50Hz)
INSTALLATION : COUNTER TOP
MAX.OPERATION TDS : 4000 PPM
MEMBRANE TYPE : THIN FILM COMPOSITE
MIN.INLET PRESSURE / TEMP : 0.3 kg / cm2, 10 °C
MODEL : WHALE 25
OPERATING VOLTAGE : 48 VOLT (DC)
PRODUCT DIMENSION : 21.1 (H) x 9.9 (W) x 16.7 (L)
PURIFICATION CAPACITY : 25 LITRES PER HOUR
RECOVERY RATE : MORE THAN 30% AT 27°c ± 2°c
SMPS : MEMBRELLA / EQUALIANT
SOLENOID VALVE : MEMBRELLA / SLX
STORAGE CAPACITY : 20 LITRES
TECHNOLOGY : REVERSE OSMOSIS SYSTEM
TOTAL POWER CONSUMPTION : 50 W
TUBE 1/4 : 5 METERS
TUBE 3/8 : 2 METERS
WEIGHT : 18 kg (Approx)
WARRENTY &  SUPPORT : Since Whale  designs its purifiers and many of its parts  are a truly integrated system. Dealer only  can provide one-stop service ,guaranty and support for any service and maintenance, so most issues can be resolved in a single visit

Answer 2

The data you are searching are actually loaded by javascript. You have to use a package such as selenium to retrieve the data.

You can try this:

CODE:

from bs4 import BeautifulSoup as bs
from selenium import webdriver
import requests
from selenium.webdriver.firefox.options import Options as FirefoxOptions

# Use options to have your selenium headless
options = FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)

url = "http://opor.in/ProductDetail/Index?ProductId=212"
driver.get(url)

page = driver.page_source
html = bs(page, 'html.parser')

model_name = html.find('div', {'class':'product-name'})
spec = html.find('div', {'class':'specification'})
print(model_name)
print(spec)

RESULTS:

<div class="product-name">WHALE 25 LPH</div>
<div class="specification">
<div>Specifications</div>
<table><tr><td class="specification-group" colspan="2"><div>General</div></td></tr><tr><td>Product Code</td><td>1601KFMB</td></tr><tr><td>MEMBRANE</td><td>MEMBRELLA -ALPHA- 80 GPD (2 NOS)</td></tr><tr><td>PUMP</td><td>KEMFLO 48 V</td></tr><tr><td class="specification-group" colspan="2"><div>Specifications</div></td></tr><tr><td>APPLICATION</td><td>SUITABLE FOR BRACKISH WATER</td></tr><tr><td>FILTER LIFE</td><td>APPROX 3000 LITRE / 6 MONTHS</td></tr><tr><td>FILTERS</td><td>SEDIMENT, PRECARBON, POST CARBON</td></tr><tr><td>FLOAT</td><td>MEMBRELLA</td></tr><tr><td>FR</td><td>MEMBRELLA /KFL</td></tr><tr><td>INLINE SET</td><td>MEMBRELLA</td></tr><tr><td>INPUT VOLTAGE</td><td>100-300 VOLT AC (50Hz)</td></tr><tr><td>INSTALLATION</td><td>COUNTER TOP</td></tr><tr><td>MAX.OPERATION TDS</td><td>4000 PPM</td></tr><tr><td>MEMBRANE TYPE</td><td>THIN FILM COMPOSITE</td></tr><tr><td>MIN.INLET PRESSURE / TEMP</td><td>0.3 kg / cm2, 10 °C</td></tr><tr><td>MODEL</td><td>WHALE 25</td></tr><tr><td>OPERATING VOLTAGE</td><td>48 VOLT (DC)</td></tr><tr><td>PRODUCT DIMENSION</td><td>21.1 (H) x 9.9 (W) x 16.7 (L)</td></tr><tr><td>PURIFICATION CAPACITY</td><td>25 LITRES PER HOUR</td></tr><tr><td>RECOVERY RATE</td><td>MORE THAN 30% AT 27°c ± 2°c</td></tr><tr><td>SMPS</td><td>MEMBRELLA / EQUALIANT</td></tr><tr><td>SOLENOID VALVE</td><td>MEMBRELLA / SLX</td></tr><tr><td>STORAGE CAPACITY</td><td>20 LITRES</td></tr><tr><td>TECHNOLOGY</td><td>REVERSE OSMOSIS SYSTEM</td></tr><tr><td>TOTAL POWER CONSUMPTION</td><td>50 W</td></tr><tr><td>TUBE 1/4</td><td>5 METERS</td></tr><tr><td>TUBE 3/8</td><td>2 METERS</td></tr><tr><td>WEIGHT</td><td>18 kg (Approx)</td></tr><tr><td>WARRENTY &amp;  SUPPORT</td><td>Since Whale  designs its purifiers and many of its parts  are a truly integrated system. Dealer only  can provide one-stop service ,guaranty and support for any service and maintenance, so most issues can be resolved in a single visit</td></tr></table>
</div>