[英]I am trying to scrape a website using python request that doesn't change its link when click on load more i tried but looks its json
我正在嘗試從網站上抓取所有添加鏈接，並且我是否成功地這樣做了。 請檢查鏈接https://www.olx.in/hyderabad_g4058526/q-note-9-pro?isSearchCall=true
網站頁面有一個加載更多按鈕，用於加載更多添加。 [1]： https://ibb.co/DGNsG6j"
問題是單擊加載更多不會更改頁面的 URL，因此我只能抓取默認顯示的初始鏈接請幫助我。
您是否考慮過使用 Selenium？ 它能夠通過定位 HTML 標簽並點擊鏈接來自動點擊網站。 https://selenium-python.readthedocs.io/index.html
查看他們的私有 API； 檢查元素和網絡選項卡。 請注意，當您單擊“加載更多”按鈕時，您的瀏覽器會向以下 URL 發出請求：
https://www.olx.in/api/relevance/v2/search?facet_limit=100&lang=en&latitude=17.46497&location=4058526&location_facet_limit=20&longitude=78.43517&page=1&platform=web-desktop&query=note%209%20pro&spellcheck=true
您可以通過 URL 參數修改查詢。 然后，使用requests ，您可以查詢此信息並組織 JSON 返回：
{
"version": "100.0",
"data": [
{
"id": "1636448838",
"score": 1679.043212890625,
"spell": {
"id": 54,
"key": "GEAS",
"version": "1",
"main": true,
"facet_disabled": false,
"default_sorting": "DEFAULT"
},
"status": {
"status": "active",
"allow_edit": true,
"ban_reason_id": null,
"display": "active",
"translated_display": "Active",
"link": null,
"flags": {
"new": false,
"hot": false
},
"message": null
},
"category_id": "1453",
"favorites": {
"count": 0,
"count_label_next": "1"
},
"images": [
{
"id": "1003855963",
"external_id": "tj7vu6vj98qp-IN",
"width": 524,
"height": 1080,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/tj7vu6vj98qp-IN/image",
"full": {
"width": 1080,
"height": 2225,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/tj7vu6vj98qp-IN/image;s=1080x2225"
},
"big": {
"width": 505,
"height": 1040,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/tj7vu6vj98qp-IN/image;s=505x1040"
},
"medium": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/tj7vu6vj98qp-IN/image;s=120x247"
},
"small": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/tj7vu6vj98qp-IN/image;s=120x247"
}
},
{
"id": "1003855964",
"external_id": "s6z25cvcbz052-IN",
"width": 524,
"height": 1080,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/s6z25cvcbz052-IN/image",
"full": {
"width": 1080,
"height": 2225,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/s6z25cvcbz052-IN/image;s=1080x2225"
},
"big": {
"width": 505,
"height": 1040,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/s6z25cvcbz052-IN/image;s=505x1040"
},
"medium": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/s6z25cvcbz052-IN/image;s=120x247"
},
"small": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/s6z25cvcbz052-IN/image;s=120x247"
}
},
{
"id": "1003855965",
"external_id": "mq6ouyke1k3h3-IN",
"width": 524,
"height": 1080,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/mq6ouyke1k3h3-IN/image",
"full": {
"width": 1080,
"height": 2225,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/mq6ouyke1k3h3-IN/image;s=1080x2225"
},
"big": {
"width": 505,
"height": 1040,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/mq6ouyke1k3h3-IN/image;s=505x1040"
},
"medium": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/mq6ouyke1k3h3-IN/image;s=120x247"
},
"small": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/mq6ouyke1k3h3-IN/image;s=120x247"
}
},
{
"id": "1003855966",
"external_id": "7vfcbj5fb8fh2-IN",
"width": 524,
"height": 1080,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/7vfcbj5fb8fh2-IN/image",
"full": {
"width": 1080,
"height": 2225,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/7vfcbj5fb8fh2-IN/image;s=1080x2225"
},
"big": {
"width": 505,
"height": 1040,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/7vfcbj5fb8fh2-IN/image;s=505x1040"
},
"medium": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/7vfcbj5fb8fh2-IN/image;s=120x247"
},
"small": {
"width": 120,
"height": 247,
"url": "https://apollo-singapore.akamaized.net:443/v1/files/7vfcbj5fb8fh2-IN/image;s=120x247"
}
}
],
"certified_car": false,
"is_kyc_verified_user": false,
"has_phone_param": false,
"locations_resolved": {
"COUNTRY_id": "1000001",
"COUNTRY_name": "India",
"ADMIN_LEVEL_1_id": "2007599",
"ADMIN_LEVEL_1_name": "Telangana",
"ADMIN_LEVEL_3_id": "4058526",
"ADMIN_LEVEL_3_name": "Hyderabad",
"SUBLOCALITY_LEVEL_1_id": "5348837",
"SUBLOCALITY_LEVEL_1_name": "Murad Nagar"
},
"description": "Full condition with box + bill charger",
"created_at": "2021-05-18T10:53:54+05:30",
"inspection_info": null,
"package_id": null,
"title": "Note 9 pro",
"main_info": null,
"user_type": "Regular",
"display_date": "2021-05-18T05:23:54+0000",
"user_id": "508731436",
"price": {
"value": {
"raw": 14500,
"currency": {
"iso_4217": "INR",
"pre": "₹"
},
"display": "₹ 14,500"
},
"key_name": "Price",
"key": "price"
},
"created_at_first": "2021-05-18T10:53:54+05:30",
"locations": [
{
"lat": 17.39,
"lon": 78.448,
"region_id": "2007599",
"district_id": "5348837",
"city_id": "4058526"
}
],
"parameters": [
{
"type": "single",
"key": "make",
"value": "mi-phone",
"key_name": "Brand",
"formatted_value": "Mi",
"value_name": "Mi"
}
],
"monetizationInfo": null
}
],
"metadata": {
"sections": [
{
"id": 10,
"offset": 0,
"criterion": "less_than",
"distance": 10,
"relaxation_type": "bucket"
},
{
"id": 20,
"offset": 39,
"criterion": "less_than",
"distance": 20,
"relaxation_type": "bucket"
}
],
"feed_version": "100.0",
"total_ads": 190,
"total_suggested_ads": 0,
"ads_on_page": 20,
"total_pages": 4,
"suggested_sections": [],
"original_term": "note 9 pro",
"modified_term": "note 9 pro",
"original_label": "Showing results for ${original_term}",
"show_hint": false,
"show_original_items": true,
"show_suggested_items": false,
"filters": [
{
"id": "price",
"values": [],
"display_order": 200,
"description": "Budget",
"range": [
{
"id": "min",
"description": "min",
"min_value": 0,
"max_value": 10000000000
},
{
"id": "max",
"description": "max",
"min_value": 0,
"max_value": 10000000000
}
],
"type": "range-input",
"render_as": "range-input"
}
],
"applied_sorting": {
"key": "desc-creation"
},
"next_page_url": "http://api.olx.in/relevance/v2/search?facet_limit=100&clientId=pwa&location_facet_limit=20&spellcheck=true&latitude=17.46497&query=note%209%20pro&location=4058526&page=2&lang=en&clientVersion=7.10.1&platform=web-desktop&longitude=78.43517",
"applied_filters": [],
"search_query": "note 9 pro",
"modified_filters": {}
},
"empty": false,
"not_empty": true,
"suggested_data": []
}
我刪除了除上述項目之外的所有項目條目。
當使用請求和beautifulsoup加載更多內容時，我無法抓取下一頁上url不會更改的網站
[英]I can't scrape a website where url not change on its next page when load more using requests and beautifulsoup
我正在嘗試抓取網站，但遇到了問題。 當我嘗試抓取數據時，看起來 html 與我看到的不同
[英]I am trying to scrape website, but I encountered a problem. When I try to scrape data, it looks like the html differs from what I see on
僅從網站獲取 JSON 的一部分，我正在嘗試使用 Python、BeautifulSoup、請求來抓取。 從 62 條回復中獲得 20 條回復
[英]Only getting a portion of JSON from website I am trying to scrape using Python, BeautifulSoup, Requests. Getting 20 responses out of 62
我正在嘗試瀏覽網站的頁面並抓取其鏈接，但即使在更改頁碼后也會抓取相同的頁面數據
[英]I am trying to navigate through the pages of a website and scrape its links but the same page data is scraped even after changing page number
我正在嘗試通過網頁抓取一個網站，當我試圖將其轉換為一個csv文件時，數據不會進入正確的列
[英]I am trying to web scrape a website and when I am trying to turn this into a csv file, data isn't going to the correct column
我試圖 web 刮取福布斯業務，但是當我請求 url 它沒有給我正確的 json 數據
[英]Im trying to web scrape Forbes business but when i request the url it doesn't give me the correct json data
我應該在給定結構中使用哪種回歸...看起來像邏輯回歸，但是當我嘗試不工作時
[英]Which regression should i use for the give structure … its looks like logistic regression but when i tried not working
我正在嘗試使用 beautifulsoup4 抓取網站並請求庫
[英]I am trying to scrape a website using beautifulsoup4 and requests library
我正在嘗試使用下面的代碼找到“圖像”按鈕，然后使用 Pycharm 中的 python 單擊該按鈕。但它顯示了一些錯誤
[英]I am trying this below code to find the 'Images' button and click on that using python in Pycharm. But its showing some errors
我正在嘗試使用 request 和 bs4 python 模塊從網站提取數據。 當我嘗試訪問此代碼時，收到以下錯誤消息給 json
[英]I am trying extract data from website using request and bs4 python module. when i try to access this code, got below error message to json
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.