如何从网站中提取 URL？

Question

I am trying to get the list of URLs of each restaurant from this website . So far this is the code that I am trying to implement

Reproducible example

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver_path = '/Users/driverpath'

from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))


driver.get('https://btownmenus.com/some/bloomington/delivery/all')

cards = driver.find_elements(By.CSS_SELECTOR, "div.restaurant-info")
urls = [card.get_attribute('href') for card in cards]
URLs

Result

I am getting None as result and I wonder if I am selecting in the wrong way the CSS of the div.

Answer 1

You need to get the a element inside the div with the restaurant-info class before you can get the href attribute from it:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://btownmenus.com/some/bloomington/delivery/all")

infos = driver.find_elements(By.CLASS_NAME, "restaurant-info")
for info in infos:
    print(info.find_element(By.CSS_SELECTOR, "a").get_attribute("href"))

Output:

https://btownmenus.com/restaurants/stir-coffee/delivery
https://btownmenus.com/restaurants/avers-north-o449/delivery
https://btownmenus.com/restaurants/wings-x-extended-delivery/delivery
https://btownmenus.com/restaurants/the-chocolate-moose/delivery
https://btownmenus.com/restaurants/gudon/delivery
https://btownmenus.com/restaurants/do-asian-fusion-restaurant/delivery
https://btownmenus.com/restaurants/wings-xtreme-o263/delivery
https://btownmenus.com/restaurants/avers-pizza-south-o465/delivery
https://btownmenus.com/restaurants/papa-johns-south-o476/delivery
https://btownmenus.com/restaurants/white-castle/delivery
https://btownmenus.com/restaurants/taco-bell/delivery
https://btownmenus.com/restaurants/mcdonalds/delivery
https://btownmenus.com/restaurants/india-garden/delivery
https://btownmenus.com/restaurants/brusters-ice-cream/delivery
https://btownmenus.com/restaurants/my-thai-cafeplus/delivery
https://btownmenus.com/restaurants/da-vinci-pizza-pasta/delivery
https://btownmenus.com/restaurants/rush-bowls/delivery
https://btownmenus.com/restaurants/subway-r5626/delivery
https://btownmenus.com/restaurants/the-chef-s-table/delivery
https://btownmenus.com/restaurants/dk-sweets/delivery
https://btownmenus.com/restaurants/cafe-bali/delivery
https://btownmenus.com/restaurants/sweet-escape-btown/delivery
https://btownmenus.com/restaurants/rush-hour-station-r8330/delivery
https://btownmenus.com/restaurants/potbelly-sandwich-shop/delivery
https://btownmenus.com/restaurants/mr-pot/delivery
https://btownmenus.com/restaurants/in-bloom-eats-juice/delivery
https://btownmenus.com/restaurants/fazolis/delivery
https://btownmenus.com/restaurants/my-happy-thai/delivery
https://btownmenus.com/restaurants/chop-shop/delivery
https://btownmenus.com/restaurants/irish-lion-c182/delivery
https://btownmenus.com/restaurants/social-cantina/delivery
https://btownmenus.com/restaurants/apna-bazaar/delivery
https://btownmenus.com/restaurants/zero-degree-snow-ice/delivery
https://btownmenus.com/restaurants/dats/delivery
https://btownmenus.com/restaurants/bloomingfoods-near-west-side/delivery
https://btownmenus.com/restaurants/red/delivery
https://btownmenus.com/restaurants/buccetos-west/delivery
https://btownmenus.com/restaurants/amrit-india-restaurant/delivery
https://btownmenus.com/restaurants/subway-w-bloomfield-rd/delivery
https://btownmenus.com/restaurants/dagwood-s-deli/delivery
https://btownmenus.com/restaurants/chow-bar-c4355/delivery
https://btownmenus.com/restaurants/mura-sushi/delivery
https://btownmenus.com/restaurants/sunny-poke/delivery
https://btownmenus.com/restaurants/burma-garden/delivery
https://btownmenus.com/restaurants/feta-kitchen-cafe/delivery
https://btownmenus.com/restaurants/laughing-planet-cafe/delivery
https://btownmenus.com/restaurants/sunny-palace/delivery
https://btownmenus.com/restaurants/gourmet-garden/delivery
https://btownmenus.com/restaurants/best-taste/delivery
https://btownmenus.com/restaurants/krispy-krunchy-chicken-r2118/delivery
https://btownmenus.com/restaurants/honey-baked-ham/delivery
https://btownmenus.com/restaurants/kilroy-s-on-kirkwood/delivery
https://btownmenus.com/restaurants/indian-palace/delivery
https://btownmenus.com/restaurants/longfei-chinese-restaurant/delivery
https://btownmenus.com/restaurants/crazy-horse/delivery
https://btownmenus.com/restaurants/nourish-bar/delivery
https://btownmenus.com/restaurants/anatolia-turkish-cuisine-r65/delivery
https://btownmenus.com/restaurants/lotus-garden/delivery
https://btownmenus.com/restaurants/jersey-mike-s-subs-r296/delivery
https://btownmenus.com/restaurants/the-3-amigos/delivery
https://btownmenus.com/restaurants/swing-in-pizza/delivery
https://btownmenus.com/restaurants/subway/delivery
https://btownmenus.com/restaurants/golden-china/delivery
https://btownmenus.com/restaurants/hartzell-s-ice-cream/delivery
https://btownmenus.com/restaurants/village-pub/delivery
https://btownmenus.com/restaurants/On-theway/delivery
https://btownmenus.com/restaurants/scholars-inn-bakehouse/delivery
https://btownmenus.com/restaurants/noodle-town-r8354/delivery
https://btownmenus.com/restaurants/anyetsangs-little-tibet-o1325/delivery
https://btownmenus.com/restaurants/el-ranchero-mexican-restaurant/delivery
https://btownmenus.com/restaurants/my-thai-cafe-sushi/delivery
https://btownmenus.com/restaurants/sakura-15/delivery
https://btownmenus.com/restaurants/burgers-wings-things-o736/delivery
https://btownmenus.com/restaurants/korea-restaurant/delivery
https://btownmenus.com/restaurants/naughty-dog/delivery
https://btownmenus.com/restaurants/fat-dan-s-chicago-deli/delivery
https://btownmenus.com/restaurants/red-mango/delivery
https://btownmenus.com/restaurants/mr-taco/delivery
https://btownmenus.com/restaurants/sushi-bar-r1501/delivery
https://btownmenus.com/restaurants/szechuan-kitchen/delivery
https://btownmenus.com/restaurants/popkorn-twist/delivery
https://btownmenus.com/restaurants/smokeworks/delivery
https://btownmenus.com/restaurants/noble-roman-s-pizza/delivery
https://btownmenus.com/restaurants/btown-gyros/delivery
https://btownmenus.com/restaurants/viva-mas-mexican-restaurant/delivery
https://btownmenus.com/restaurants/szechuan-taste/delivery
https://btownmenus.com/restaurants/little-downtown-cafe/delivery
https://btownmenus.com/restaurants/mr-hibachi/delivery
https://btownmenus.com/restaurants/rockits-pizza-o616/delivery
https://btownmenus.com/restaurants/which-wich-superior-sandwiches-o1769/delivery
https://btownmenus.com/restaurants/subway-r1047/delivery
https://btownmenus.com/restaurants/the-cabin-restaurant/delivery
https://btownmenus.com/restaurants/sobon-korean-cafe/delivery
https://btownmenus.com/restaurants/buccetos-o103/delivery
https://btownmenus.com/restaurants/smokin-jacks/delivery
https://btownmenus.com/restaurants/deangelos-c5671/delivery
https://btownmenus.com/restaurants/brilliant-coffee-company/delivery
https://btownmenus.com/restaurants/apna-grocery-convenience/delivery
https://btownmenus.com/restaurants/mama-s-korean-bbq/delivery
https://btownmenus.com/restaurants/bangkok-thai-cuisine/delivery
https://btownmenus.com/restaurants/subway-r1979/delivery
https://btownmenus.com/restaurants/roly-poly/delivery
https://btownmenus.com/restaurants/denny-s/delivery
https://btownmenus.com/restaurants/bloomingfood-s-ivy-tech/delivery
https://btownmenus.com/restaurants/el-ranchero-mexican-restaurant-r2101/delivery
https://btownmenus.com/restaurants/butchs-grillacatessen-and-eatzeria-o725/delivery
https://btownmenus.com/restaurants/bedrak-cafe/delivery
https://btownmenus.com/restaurants/el-ranchero-mexican-restaurant-r3334/delivery
https://btownmenus.com/restaurants/trailhead-pizzeria/delivery
https://btownmenus.com/restaurants/the-tap/delivery
https://btownmenus.com/restaurants/taste-of-india-o797/delivery
https://btownmenus.com/restaurants/bloomington-sandwich-co-o787/delivery
https://btownmenus.com/restaurants/avers-pizza-east-o1737/delivery
https://btownmenus.com/restaurants/peach-garden-o45/delivery
https://btownmenus.com/restaurants/hive/delivery
https://btownmenus.com/restaurants/the-orbit-room/delivery
https://btownmenus.com/restaurants/eric-gordon-s-greek-s-pizzeria/delivery
https://btownmenus.com/restaurants/hoosiers-pizza/delivery
https://btownmenus.com/restaurants/the-bowl/delivery
https://btownmenus.com/restaurants/brown-diner/delivery
https://btownmenus.com/restaurants/buffalo-wild-wings-r2144/delivery
https://btownmenus.com/restaurants/chick-fil-a/delivery
https://btownmenus.com/restaurants/subway-r6058/delivery
https://btownmenus.com/restaurants/the-trojan-horse/delivery
https://btownmenus.com/restaurants/yogi-s-bar-grill/delivery
https://btownmenus.com/restaurants/bloomingfoods-east/delivery
https://btownmenus.com/restaurants/asuka-r1874/delivery
https://btownmenus.com/restaurants/restaurant-ami-c4353/delivery
https://btownmenus.com/restaurants/japonee-express/delivery
https://btownmenus.com/restaurants/sofra-cafe-r6465/delivery
https://btownmenus.com/restaurants/buffalouies-o54/delivery
https://btownmenus.com/restaurants/freddy-s-frozen-custard/delivery
https://btownmenus.com/restaurants/chipotle-mexican-grill/delivery
https://btownmenus.com/restaurants/five-guys/delivery
https://btownmenus.com/restaurants/kfc/delivery
https://btownmenus.com/restaurants/bloomington-bagel-company/delivery
https://btownmenus.com/restaurants/wendy-s/delivery
https://btownmenus.com/restaurants/qdoba/delivery
https://btownmenus.com/restaurants/china-wok/delivery
https://btownmenus.com/restaurants/square-donuts/delivery
https://btownmenus.com/restaurants/siam-house/delivery
https://btownmenus.com/restaurants/z-c-teriyaki/delivery
https://btownmenus.com/restaurants/burger-king/delivery
https://btownmenus.com/restaurants/joella-s-hot-chicken/delivery
https://btownmenus.com/restaurants/applebees-c179/delivery
https://btownmenus.com/restaurants/wheel-pizza-r6622/delivery
https://btownmenus.com/restaurants/japonee-on-walnut/delivery
https://btownmenus.com/restaurants/papa-johns-north-o467/delivery