Not getting output when trying to scrape data with html module in lxml library and xpath

Question

I was trying to scrape the links of products in the Amazon Best Sellers site for the category Computers and Accessories.

When I did it with BeautifulSoup and CSS Selectors with class name 'p13n-desktop-grid' to identify the whole products grid, I got the output as shown below.

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get('https://www.amazon.com/Best-Sellers-Computers-Accessories/zgbs/pc/ref=zg_bs_pg_1?_encoding=UTF8&pg=1')
content = driver.page_source

homepage_soup = BeautifulSoup(content, 'html.parser')
all_products = homepage_soup.find('div', attrs={"class": "p13n-desktop-grid"})
for product_section in all_products.find_all('div', {'id': 'gridItemRoot'}):
    for product_link in product_section.find_all('a',{'tabindex':'-1'}):
        print(product_link['href'])

Code and output when I used CSS Selectors

But when I tried to use Requests library to get the link and html module from the lxml library and xpath instead of CSS Selectors to do the same, I am getting an empty list.

import requests
from lxml import html

page=requests.get('https://www.amazon.com/Best-Sellers-Computers-Accessories/zgbs/pc/ref=zg_bs_pg_1?_encoding=UTF8&pg=1')
tree=html.fromstring(page.content)
print(tree)
all_products=tree.xpath('//*[@id="CardInstancekNA5iTRPgXKDudcFAO3j3Q"]/div[2]/div[1]')
print(all_products)

Code and output when I used xpath

Does anyone know why is this so? Thanks in advance.

Answer 1

Here is a way of getting those products, as a dataframe:

BeautifulSoup

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

pd.set_option('display.max_rows', None, 'display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)

big_list = []
for x in range(1, 3):
    url = f'https://www.amazon.com/Best-Sellers-Computers-Accessories/zgbs/pc/ref=zg_bs_pg_{x}?_encoding=UTF8&pg={x}'
    r = s.get(url)
    soup = bs(r.text, 'html.parser')
    products = soup.select('div [class*="-uncoverable-faceout"] a[class="a-link-normal"]:nth-of-type(2)')
    for p in products:
        url = 'https://www.amazon.com' + p.get('href')
        product = p.text.strip()
        big_list.append((product, url))
df = pd.DataFrame(big_list, columns = ['Product', 'Url'])
print(df)

Result in terminal:

    Product Url
0   Amazon Fire HD 10 tablet, 10.1", 1080p Full HD, 32 GB, latest model (2021 release), Black   https://www.amazon.com/Fire-HD-10-tablet/dp/B08BX7FV5L/ref=zg_bs_pc_sccl_1/138-1927696-9589501?psc=1
1   All-new Amazon Fire HD 8 tablet, 8” HD Display, 32 GB, 30% faster processor, designed for portable entertainment, (2022 release), Black https://www.amazon.com/All-new-Fire-HD-8-tablet/dp/B099Z8HLHT/ref=zg_bs_pc_sccl_2/138-1927696-9589501?psc=1
2   Amazon Fire HD 10 Plus tablet, 10.1", 1080p Full HD, 32 GB, latest model (2021 release), Slate  https://www.amazon.com/Fire-HD-10-Plus-tablet/dp/B08F6FYN6B/ref=zg_bs_pc_sccl_3/138-1927696-9589501?psc=1
3   Amazon Fire 7 Kids tablet, 7" display, ages 3-7, with ad-free content kids love, 2-year worry-free guarantee, parental controls, 16 GB, (2022 release), Blue    https://www.amazon.com/Fire-7-Kids-Tablet/dp/B099HDR2Y6/ref=zg_bs_pc_sccl_4/138-1927696-9589501?psc=1
4   USB 3.0 Hub, VIENON 4-Port USB Hub USB Splitter USB Expander for Laptop, Xbox, Flash Drive, HDD, Console, Printer, Camera,Keyborad, Mouse   https://www.amazon.com/Extender-Splitter-MacBook-Chromebook-Pixelbook/dp/B09MLRPTT2/ref=zg_bs_pc_sccl_5/138-1927696-9589501?psc=1
5   Amazon Fire HD 10 Kids tablet, 10.1", 1080p Full HD, ages 3–7, 32 GB, Sky Blue  https://www.amazon.com/Fire-HD-10-Kids-tablet/dp/B08F62DV1Z/ref=zg_bs_pc_sccl_6/138-1927696-9589501?psc=1
6   Amazon Fire HD 10 Kids Pro tablet, 10.1", 1080p Full HD, ages 6–12, 32 GB, (2021 release), named"Best Tablet for Big Kids" by Good Housekeeping, Doodle https://www.amazon.com/Fire-HD-10-Kids-Pro-tablet/dp/B08F5LWB6R/ref=zg_bs_pc_sccl_7/138-1927696-9589501?psc=1
7   TP-Link AX1800 WiFi 6 Router (Archer AX21) – Dual Band Wireless Internet Router, Gigabit Router, USB port, Works with Alexa - A Certified for Humans Device https://www.amazon.com/WiFi-6-Router-Gigabit-Wireless/dp/B08H8ZLKKK/ref=zg_bs_pc_sccl_8/138-1927696-9589501?psc=1
8   All-new Amazon Fire HD 8 Kids tablet, 8" HD display, ages 3-7, with age-appropriate curated content and easy-to-use Parent Dashboard, 32 GB, (2022 release), Blue   https://www.amazon.com/Amazon-Official-Site-Fire-HD-Kids-8-Tablet-8-inch-Display-2022-release/dp/B09BG5LFGS/ref=zg_bs_pc_sccl_9/138-1927696-9589501?psc=1
9   4-Port USB Hub 3.0, T-Sound USB Splitter for Laptop, Ps4 Keyboard and Mouse Adapter for Dell, Asus, HP, MacBook Air, Surface Pro, Acer, Xbox, Flash Drive, Mobile HD, Console, Printer, Camera  https://www.amazon.com/T-Sound-Splitter-Keyboard-Adapter-Nintendo/dp/B083RQMC7S/ref=zg_bs_pc_sccl_10/138-1927696-9589501?psc=1
10  All-new Amazon Fire HD 8 Kids Pro tablet, 8" HD display, ages 6-12, 30% faster processor, 13 hours battery life, Kid-Friendly Case, 32 GB, (2022 release), Rainbow Universe https://www.amazon.com/Amazon-Official-Site-Fire-HD-Kids-8-Pro-Tablet-8-inch-Display-2022-release/dp/B09BG63ZMM/ref=zg_bs_pc_sccl_11/138-1927696-9589501?psc=1
11  Mac Book Pro Charger - 118W USB C Charger Power Adapter Compatible with MacBook Pro 16, 15, 14, 13 Inch, MacBook Air 13 Inch, iPad Pro and All USB C Device, 7.2ft USB C to C Cable https://www.amazon.com/Mac-Book-Pro-Charger-Compatible/dp/B09GFR7NSK/ref=zg_bs_pc_sccl_12/138-1927696-9589501?psc=1
12  Acer Aspire 5 A515-56-32DK Slim Laptop - 15.6" Full HD IPS Display - 11th Gen Intel i3-1115G4 Dual Core Processor - 4GB DDR4 - 128GB NVMe SSD - WiFi 6 - Amazon Alexa - Windows 11 Home in S mode.  https://www.amazon.com/Acer-A515-56-32DK-Display-i3-1115G4-Processor/dp/B09R5VYRVN/ref=zg_bs_pc_sccl_13/138-1927696-9589501?psc=1
13  SABRENT 4 Port USB 2.0 Data Hub with Individual LED lit Power Switches [Charging NOT Supported] for Mac & PC (HB-UMLS)  https://www.amazon.com/Sabrent-4-Port-Individual-Switches-HB-UMLS/dp/B00BWF5U0M/ref=zg_bs_pc_sccl_14/138-1927696-9589501?psc=1
14  Mac Book Pro Charger - 118W USB C Charger Power Adapter Compatible with MacBook Pro 16, 15, 14, 13 Inch, MacBook Air 13 Inch, iPad Pro 2021/2020/2019/2018, Included 7.2ft USB C to C Cable https://www.amazon.com/Mac-Book-Pro-Charger-Compatible/dp/B097PTBB5V/ref=zg_bs_pc_sccl_15/138-1927696-9589501?psc=1
15  Lenovo 2022 Newest Ideapad 3 Laptop, 15.6" HD Touchscreen, 11th Gen Intel Core i3-1115G4 Processor, 8GB DDR4 RAM, 256GB PCIe NVMe SSD, HDMI, Webcam, Wi-Fi 5, Bluetooth, Windows 11 Home, Almond    https://www.amazon.com/Lenovo-Ideapad-Touchscreen-i3-1005G1-Processor/dp/B08B6F1NNR/ref=zg_bs_pc_sccl_16/138-1927696-9589501?psc=1
16  Amazon Fire HD 8 Plus tablet, HD display, 32 GB, (2020 release), our best 8" tablet for portable entertainment, Slate   https://www.amazon.com/Fire-HD-8-Plus-tablet/dp/B0839NDRB2/ref=zg_bs_pc_sccl_17/138-1927696-9589501?psc=1
17  Amazon Fire 7 tablet, 7” display, 16 GB, 10 hours battery life, light and portable for entertainment at home or on-the-go, (2022 release), Black    https://www.amazon.com/Fire-7-Tablet/dp/B096WKKK2K/ref=zg_bs_pc_sccl_18/138-1927696-9589501?psc=1
18  SiliconV 60W Mac Book Pro Charger, Replacement for Mac Book Pro 13-Inch 2012-2016 Retina Display Ac 60W 2 T Connector Power Adapter, Laptop Charger Compatible with Mac Book Pro/Mac Book Air   https://www.amazon.com/SiliconV-Charger-Replacement-Magnetic-Connector/dp/B08JPHHV61/ref=zg_bs_pc_sccl_19/138-1927696-9589501?psc=1
19  Docking Station USB C to Dual HDMI Adapter, USB C Hub Dual HDMI Monitors for Windows,USB C Adapter with Dual HDMI,3 USB Port,PD Compatible for Dell XPS 13/15, Lenovo Yoga,etc  https://www.amazon.com/Adapter-Docking-Station-Monitors-Compatible/dp/B08CRCRPCJ/ref=zg_bs_pc_sccl_20/138-1927696-9589501?psc=1
20  Google Wifi - AC1200 - Mesh WiFi System - Wifi Router - 4500 Sq Ft Coverage - 3 pack    https://www.amazon.com/Google-Wifi-System-Router-Replacement/dp/B08GG9CMLR/ref=zg_bs_pc_sccl_21/138-1927696-9589501?psc=1
21  300 PCS Stickers Pack (50-850Pcs/Pack), Colorful VSCO Waterproof Stickers, Cute Aesthetic Stickers. Laptop, Water Bottle, Phone, Skateboard Stickers for Teens Girls Kids, Vinyl Sticker.   https://www.amazon.com/Stickers-50-500Pcs-Colorful-Waterproof-Aesthetic/dp/B08518LVFD/ref=zg_bs_pc_sccl_22/138-1927696-9589501?psc=1
22  Fire HD 8 tablet, 8" HD display, 32 GB, (2020 release), designed for portable entertainment, Black  https://www.amazon.com/Fire-HD-8-tablet/dp/B07TMJ1R3X/ref=zg_bs_pc_sccl_23/138-1927696-9589501?psc=1
23  uni USB C to USB Hub 4 Ports, Aluminum USB Type C to USB Adapter with 4 USB 3.0 Ports, Thunderbolt 3 to Multiport USB 3.0 Hub Adapter for MacBook Pro/Air 2020/2019, iPad Pro, Dell, Chromebook and more    https://www.amazon.com/uni-Aluminum-Thunderbolt-Multiport-Chromebook/dp/B087QFM3GF/ref=zg_bs_pc_sccl_24/138-1927696-9589501?psc=1
24  TP-Link Deco AX3000 WiFi 6 Mesh System(Deco X55) - Covers up to 6500 Sq.Ft. , Replaces Wireless Router and Extender, 3 Gigabit ports per unit, supports Ethernet Backhaul (3-pack)  https://www.amazon.com/WiFi-6-Mesh-System-AX3000/dp/B09PRB1MZM/ref=zg_bs_pc_sccl_25/138-1927696-9589501?psc=1
25  All-new Amazon Fire HD 8 Plus tablet, 8” HD Display, 32 GB, 30% faster processor, 3GB RAM, wireless charging, (2022 release), Gray  https://www.amazon.com/Amazon-Official-Site-Fire-HD-8-Pro-Tablet-8-inch-Display-2022-release/dp/B099Z93WD9/ref=zg_bs_pc_sccl_26/138-1927696-9589501?psc=1
26  TP-Link AX3000 WiFi 6 Router – 802.11ax Wireless Router, Gigabit, Dual Band Internet Router, Supports VPN Server and Client, OneMesh Compatible (Archer AX55)   https://www.amazon.com/TP-Link-WiFi-AX3000-Smart-Router/dp/B09G5W9R6R/ref=zg_bs_pc_sccl_27/138-1927696-9589501?psc=1
27  TP-Link AC1750 Smart WiFi Router (Archer A7) -Dual Band Gigabit Wireless Internet Router for Home, Works with Alexa, VPN Server, Parental Control, QoS  https://www.amazon.com/TP-Link-AC1750-Smart-WiFi-Router/dp/B079JD7F7G/ref=zg_bs_pc_sccl_28/138-1927696-9589501?psc=1
28  ARRIS Surfboard S33 DOCSIS 3.1 Multi-Gigabit Cable Modem | Approved for Comcast Xfinity, Cox, Spectrum & More | 1 & 2.5 Gbps Ports | 2.5 Gbps Max Internet Speeds | 4 OFDM Channels | 2 Year Warranty   https://www.amazon.com/ARRIS-Surfboard-S33-Multi-Gigabit-Ethernet/dp/B08FMSC5WZ/ref=zg_bs_pc_sccl_29/138-1927696-9589501?psc=1
29  Mac Book Pro Charger - 118W USB C Charger Power Adapter for USB C MacBook Pro 16 15 14 13 Inch, MacBook Air 13 Inch 2021 2020 2019 2018, New iPad Pro, Include Charge Cable（7.2ft/2.2m） https://www.amazon.com/Mac-Book-Pro-Charger-Cable%EF%BC%887-2ft/dp/B09FSCMR14/ref=zg_bs_pc_sccl_30/138-1927696-9589501?psc=1
30  Amazon eero 6+ mesh Wi-Fi system | Fast and reliable gigabit speeds | connect 75+ devices | Coverage up to 4,500 sq. ft. | 3-pack, 2022 release https://www.amazon.com/eero-reliable-gigabit-connect-Coverage/dp/B08ZK2BHP2/ref=zg_bs_pc_sccl_1/138-1927696-9589501?psc=1
31  Mac Book Air Charger, Replacement 45W T-Tip AC Power Adapter Charger for Charging Mac Book Air 11-inch and 13-inch  https://www.amazon.com/Charger-Replacement-Adapter-Charging-11-inch/dp/B0B38F3739/ref=zg_bs_pc_sccl_2/138-1927696-9589501?psc=1
32  96W USB C Charger for MacBook Pro 16, 15, 14, 13 inch 2021, 2020, 2019, 2018, New MacBook Air, iPad Pro, USBC, Type C Thunderbolt Laptop Power Adapter Supply, LED, 6.6ft 5A USB C to C Charging Cable  https://www.amazon.com/Charger-Adapter-MacBook-Thunderbolt-Indicator/dp/B0837N9LVF/ref=zg_bs_pc_sccl_3/138-1927696-9589501?psc=1
33  New Genuine Lenovo ThinkPad 65 Watt 20V 3.25A Type-C USB AC Adapter ADLX65YDC2A https://www.amazon.com/Genuine-Lenovo-ThinkPad-Adapter-ADLX65YDC2A/dp/B07CGP4SB2/ref=zg_bs_pc_sccl_4/138-1927696-9589501?psc=1
34  IdeaPad Laptop Charger 65W 45W AC Adapter for Lenovo Laptop 110 110s 120s 130s 310 330S 320 330 510 510s 520 710s Yoga 710 Flex 14 14iwl Laptop Power Supply Cord…… https://www.amazon.com/IdeaPad-Laptop-Charger-Adapter-Lenovo/dp/B08PBQK7H7/ref=zg_bs_pc_sccl_5/138-1927696-9589501?psc=1
35  USB Type C 45W Chromebook Charger for HP Chromebook 14 X360 14-ca051wm 14-ca052wm 14-ca091wm ca061dx 14-ca0061dx 14-ca060nr 14-ca020nr 14-ca043cl USB C Power Supply Adapter    https://www.amazon.com/Chromebook-14-ca051wm-14-ca052wm-14-ca091wm-14-ca0061dx/dp/B09LYRT2GL/ref=zg_bs_pc_sccl_6/138-1927696-9589501?psc=1
36  [Upgraded Version] Surface Pro Charger 65W for Surface Pro 3/4/5/6/7 Power Supply Adapter, Compatible for Both Microsoft Surface Book Laptop/Tablet，Works with 65W&44W&36W&24W (6.6 Ft Cord)    https://www.amazon.com/GSNOW-Adapter-Charger-Microsoft-Charging/dp/B07RT8Y122/ref=zg_bs_pc_sccl_7/138-1927696-9589501?psc=1
37  Mac Book Pro Charger - 96W USB C Fast Charger Power Adapter Compatible with New MacBook Air 13 Inch & MacBook Pro 16, 15, 14, 13 Inch 2021 2020 2019 2018, 6.6ft USB C to C Charge Cable    https://www.amazon.com/HJCDMJ-Charger-Replacement-Compatible-2018-2020/dp/B08RYXFQDT/ref=zg_bs_pc_sccl_8/138-1927696-9589501?psc=1
38  45W USB C Type C Chromebook Charger Laptop Fast Charging Power Adapter, Widely Compatible with HP, Dell, Lenovo, Acer, Asus, Samsung, Google, and More  https://www.amazon.com/45W-Type-Charger-Power-Adapter/dp/B09MFCGJ73/ref=zg_bs_pc_sccl_9/138-1927696-9589501?psc=1
39  Lenovo IdeaPad 3 11 Chromebook Laptop,11.6" HD Display,Intel Celeron N4020, 4GB RAM, 64GB Storage, UHD Graphics 600, Chrome OS, Onyx Black  https://www.amazon.com/Lenovo-Chromebook-Processor-Graphics-82BA0003US/dp/B087YW8FQB/ref=zg_bs_pc_sccl_10/138-1927696-9589501?psc=1
40  JSAUX Docking Station Compatible with Steam Deck, 6-in-1 Steam Deck Dock with HDMI 2.0 4K@60Hz, Gigabit Ethernet, 3 USB-A 3.0 and Full Speed Charging USB-C Port Compatible with Valve Steam Deck-HB0603    https://www.amazon.com/JSAUX-Compatible-Ethernet-Charging-Deck-HB0603/dp/B0B7HVZNMB/ref=zg_bs_pc_sccl_11/138-1927696-9589501?psc=1
41  Lenovo - 2022 - IdeaPad Flex 5i - 2-in-1 Chromebook Laptop Computer - Intel Core i3-1115G4 - 13.3" FHD Touch Display - 8GB Memory - 128GB Storage - Chrome OS   https://www.amazon.com/Lenovo-2-1-Chromebook-Computer/dp/B09RNCN3YV/ref=zg_bs_pc_sccl_12/138-1927696-9589501?psc=1
42  PowerSource 19.5V 65W 45W UL Listed 14Ft Long HP Smart Blue Tip AC Adapter for Many Models Including: X360 Pavilion, Envy, Spectre, Elitebook 840, ProBook, and More Laptop Power-Supply Charger Cord   https://www.amazon.com/PowerSource-Listed-Adapter-Models-Including/dp/B07SW89GC3/ref=zg_bs_pc_sccl_13/138-1927696-9589501?psc=1
43  NETGEAR Cable Modem Wi-Fi Router Combo C6250 - Compatible with All Cable Providers Including Xfinity by Comcast, Spectrum, Cox | for Cable Plans Up to 300 Mbps | AC1600 Wi-Fi Speed | DOCSIS 3.0   https://www.amazon.com/NETGEAR-Cable-Modem-Router-Combo/dp/B0787TRNMH/ref=zg_bs_pc_sccl_14/138-1927696-9589501?psc=1
44  Chromebook Charger, USB C Laptop Charger, 45W 65W Fast Charging for Dell Samsung HP Asus Acer Google Chromebook Charger, fit USB Computer Chargers for Laptops, Type c chromebook Charger   https://www.amazon.com/Chromebook-USB-Charging-Computer-chromebook/dp/B0B762G2V5/ref=zg_bs_pc_sccl_15/138-1927696-9589501?psc=1
45  TP-Link Deco WiFi 6 Mesh System(Deco X20) - Covers up to 5800 Sq.Ft. , Replaces Wireless Routers and Extenders(3-Pack, 6 Ethernet Ports in total, supports Wired Ethernet Backhaul) https://www.amazon.com/Mesh-Wifi-6-Router-System/dp/B085Z35GY6/ref=zg_bs_pc_sccl_16/138-1927696-9589501?psc=1
46  Amazon eero 6 dual-band mesh Wi-Fi 6 extender - expands existing eero network   https://www.amazon.com/Eero-6-Extender/dp/B085VQVPJK/ref=zg_bs_pc_sccl_17/138-1927696-9589501?psc=1
47  45W 19.5V 2.31A AC Adapter Laptop Charger Compatible for HP Notebook 15 Charger 15-ba009dx 15-ba079dx 15-ba113cl 15-bs015dx 15-bs113dx 15-bs115dx 15-bw011dx 15-bw032wm Laptop PC Power Supply Cord https://www.amazon.com/Replacement-15-BS015DX-15-AF131DX-15-F272WM-15-F233WM-15-AY009DX/dp/B07MGTL6L9/ref=zg_bs_pc_sccl_18/138-1927696-9589501?psc=1
48  7-Port USB 3.0 Hub, IVETTO Data USB Hub Splitter with 3.3ft Long Cable for Laptop, PC, MacBook, Mac Pro, Mac Mini, iMac, Surface Pro and More   https://www.amazon.com/IVETTO-Individual-Switches-MacBook-Surface/dp/B08R86STK4/ref=zg_bs_pc_sccl_19/138-1927696-9589501?psc=1
49  havit HV-F2056 15.6"-17" Laptop Cooler Cooling Pad - Slim Portable USB Powered (3 Fans), Black/Blue https://www.amazon.com/HV-F2056-15-6-17-Laptop-Cooler-Cooling/dp/B00NNMB3KS/ref=zg_bs_pc_sccl_20/138-1927696-9589501?psc=1
50  Amazon eero mesh WiFi system – router replacement for whole-home coverage (3-pack)  https://www.amazon.com/Introducing-eero-mesh-WiFi-system-3-pack-/dp/B07WMLPSRL/ref=zg_bs_pc_sccl_21/138-1927696-9589501?psc=1
51  Surface Pro Charger -98W Super Fast Charger for Surface Pro 7/6/5/4/3 Power Supply Adapter, Compatible with Microsoft Surface Book Laptop/Tablet，Works with 65W&44W&36W&24W(6.8 Ft Cord)…   https://www.amazon.com/Upgraded-Version-Compatible-Microsoft-Tablet%EF%BC%8CWorks/dp/B09BJFHQYJ/ref=zg_bs_pc_sccl_22/138-1927696-9589501?psc=1
52  Bekayshad Stickers for Water Bottles, 100 Pack/PCS Hydroflask Stickers for Kids Teens Waterproof Cute Vsco Vinyl Stickers Laptop Skateboard Luggage Computer Stickers for Teens Girls Kids  https://www.amazon.com/Stickers-Hydroflask-Aesthetic-Waterproof-Skateboard/dp/B08D3GMQR6/ref=zg_bs_pc_sccl_23/138-1927696-9589501?psc=1
53  Apple 35W Dual USB-C Port Compact Power Adapter https://www.amazon.com/Apple-USB-C-Compact-Power-Adapter/dp/B0B3C493HH/ref=zg_bs_pc_sccl_24/138-1927696-9589501?psc=1
54  Motorola MG7700 Modem WiFi Router Combo with Power Boost | Approved by Comcast Xfinity, Cox and Spectrum | for Cable Plans Up to 800 Mbps | DOCSIS 3.0 + Gigabit Router https://www.amazon.com/MOTOROLA-MG7700-AC1900-Gigabit-Maximum/dp/B07BRZ2KW5/ref=zg_bs_pc_sccl_25/138-1927696-9589501?psc=1
55  Mac Book Pro Charger, Replacement for Mac Book Pro 13 Inch Display AC 60W Magnetic Shape Connector Power Adapter（Before Mid 2012）   https://www.amazon.com/Charger-Replacement-Magnetic-Connector-Adapter%EF%BC%88Before/dp/B09VX166NF/ref=zg_bs_pc_sccl_26/138-1927696-9589501?psc=1
56  Charger for Lenovo Laptop Charger - 65W 45W https://www.amazon.com/IdeaPad-Laptop-Charger-Lenovo-Adapter/dp/B08X4M4LBR/ref=zg_bs_pc_sccl_27/138-1927696-9589501?psc=1
57  USB C Docking Station Dual Monitor for Dell/HP/Lenovo/Surface Laptop, 14 in 1 Triple Display USB C Hub Multiple Adapter, USB C Dongle with 2 HDMI 4K+VGA+5 USB Port+100W PD Charger+Ethernet+SD/TF+Audio    https://www.amazon.com/Docking-Station-Monitor-Multiple-Ethernet/dp/B09WDCH35N/ref=zg_bs_pc_sccl_28/138-1927696-9589501?psc=1
58  45W 19.5V 2.31A Ac Adapter Laptop Charger for HP Pavilion x360 Charger 15-f272wm 15-f387wm 15-f233wm 15-f222wm 15-f211wm 15-f337wm 17-g121wm 17-g119dx Laptop Notebook Power Supply Cord Plug   https://www.amazon.com/15-f272wm-15-f387wm-15-f233wm-15-f222wm-15-f211wm/dp/B07H4GKHJ1/ref=zg_bs_pc_sccl_29/138-1927696-9589501?psc=1
59  Plugable USB 3.0 Universal Laptop Docking Station Dual Monitor for Windows and Mac (Dual Video: HDMI and DVI/VGA/HDMI, Gigabit Ethernet, Audio, 6 USB Ports)    https://www.amazon.com/Plugable-Universal-Laptop-Docking-Station-Dual-Monitor-2-Hdmi/dp/B00ECDM78E/ref=zg_bs_pc_sccl_30/138-1927696-9589501?psc=1

And here you can find BeautifulSoup documentation .

LXML

import requests
from lxml import html
import pandas as pd

pd.set_option('display.max_rows', None, 'display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)

big_list = []
for x in range(1, 3):
    url = f'https://www.amazon.com/Best-Sellers-Computers-Accessories/zgbs/pc/ref=zg_bs_pg_{x}?_encoding=UTF8&pg={x}'
    r = s.get(url)
    tree=html.fromstring(r.text)
    products = tree.xpath('//a[@class="a-link-normal"]')
    for p in products:
        url = 'https://www.amazon.com' + p.get('href')
        product = ''.join(p.itertext())
        if len(product) > 5 and not 'offer' in product and not 'stars' in product:
            big_list.append((product, url))
df = pd.DataFrame(big_list, columns = ['Product', 'Url'])
print(df)

Result in terminal:

Product Url
0   Amazon Fire HD 10 tablet, 10.1", 1080p Full HD, 32 GB, latest model (2021 release), Black   https://www.amazon.com/Fire-HD-10-tablet/dp/B08BX7FV5L/ref=zg_bs_pc_sccl_1/142-4412420-4348447?psc=1
1   All-new Amazon Fire HD 8 tablet, 8” HD Display, 32 GB, 30% faster processor, designed for portable entertainment, (2022 release), Black https://www.amazon.com/All-new-Fire-HD-8-tablet/dp/B099Z8HLHT/ref=zg_bs_pc_sccl_2/142-4412420-4348447?psc=1
2   Amazon Fire HD 10 Plus tablet, 10.1", 1080p Full HD, 32 GB, latest model (2021 release), Slate  https://www.amazon.com/Fire-HD-10-Plus-tablet/dp/B08F6FYN6B/ref=zg_bs_pc_sccl_3/142-4412420-4348447?psc=1
3   Amazon Fire 7 Kids tablet, 7" display, ages 3-7, with ad-free content kids love, 2-year worry-free guarantee, parental controls, 16 GB, (2022 release), Blue    https://www.amazon.com/Fire-7-Kids-Tablet/dp/B099HDR2Y6/ref=zg_bs_pc_sccl_4/142-4412420-4348447?psc=1
4   USB 3.0 Hub, VIENON 4-Port USB Hub USB Splitter USB Expander for Laptop, Xbox, Flash Drive, HDD, Console, Printer, Camera,Keyborad, Mouse   https://www.amazon.com/Extender-Splitter-MacBook-Chromebook-Pixelbook/dp/B09MLRPTT2/ref=zg_bs_pc_sccl_5/142-4412420-4348447?psc=1
5   Amazon Fire HD 10 Kids tablet, 10.1", 1080p Full HD, ages 3–7, 32 GB, Sky Blue  https://www.amazon.com/Fire-HD-10-Kids-tablet/dp/B08F62DV1Z/ref=zg_bs_pc_sccl_6/142-4412420-4348447?psc=1
6   TP-Link AX1800 WiFi 6 Router (Archer AX21) – Dual Band Wireless Internet Router, Gigabit Router, USB port, Works with Alexa - A Certified for Humans Device https://www.amazon.com/WiFi-6-Router-Gigabit-Wireless/dp/B08H8ZLKKK/ref=zg_bs_pc_sccl_7/142-4412420-4348447?psc=1
[...]

Relevant documentation for lxml can be found here .