I'm Python beginner developer, I'm still in the learning phase. More specifically working on scraping using requests and bs4. When tried to scrape the following link: ' http://directorybtr.az.gov/listings/FirmSearchResults.asp?Zip%20Like%20%22850%25%22 '
I used the following code :
import requests
from bs4 import BeautifulSoup
url ="http://directorybtr.az.gov/listings/FirmSearchResults.asp?Zip%20Like%20%22850%25%22"
res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')
res.close()
results = soup.find('table')
There is no table in results although the table are present when inspecting the source page in Chrome. Any solution or explanation please?
Thank you
Table data is inside frame, u need to go first
import requests
from lxml import html
from bs4 import BeautifulSoup
BASE_URL = "http://directorybtr.az.gov/listings/"
URL = BASE_URL + "FirmSearchResults.asp?Zip%20Like%20%22850%25%22"
#u need session because the frame use the search results data, u cant directly go to Firms.asp
session = requests.session()
response = session.get(URL)
soup = BeautifulSoup(response.text, 'lxml')
#find the first frame
frame = soup.find("frame")
#go to the frame link ( Firms.asp )
response = session.get(BASE_URL + frame.attrs['src'])
soup = BeautifulSoup(response.text, 'lxml')
table = soup.find("table")
print table
response.close()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.