如何使用两个下拉菜单在网络上抓取地图？

Question

I am new to webscraping. I have been trying to extract the business information such as ShopName and the business Address. There are two drop down menus. One correspond to the province and the other one correspond to the district. I went to the network tab inside the developer tools. Once I selected the respective province and district,I get a JSON format response for which I tested a code to extract the shopName and Info.Here's the code for the first province and first district:

import requests


headers = {
    'Connection': 'keep-alive',
    'Accept': '*/*',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36',
    'X-Requested-With': 'XMLHttpRequest',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://xn--b3cuh3bhdeppad0as7a5dybu5qd4a3kl8e.moc.go.th/MarkerShop',
    'Accept-Language': 'en-US,en;q=0.9,hi;q=0.8,bg;q=0.7,sv;q=0.6',
}

params = (
    ('prov', '81'),
    ('amphur', '8101'),
)

response = requests.get('https://xn--b3cuh3bhdeppad0as7a5dybu5qd4a3kl8e.moc.go.th/api/shopapi/getshopicon', headers=headers, params=params)


master_list=[]
for detail in details:
    data_dict={}
    invalid_tags = ['\\r', '\\n', '<', '>', '-', ';','<br>','<b>','</b>','/b',"img src='icon/icon_y.png'"," ","brb","imgsrc='icon/icon_g.png'"]
    for invalid_tag in invalid_tags:
        detail['Address']  = detail['Address'].replace(invalid_tag, '')
        detail['ShopName'] = detail['ShopName'].replace(invalid_tag,'')
    data_dict["Address_new"]=detail['Address']
    data_dict['Shop']=detail['ShopName']
    master_list.append(data_dict)

The above code gives me the required output for the first province and the first district. I want to loop over the rest of the province and their respective districts. I have been trying to find out answers. What I see is that people have using selenium to go through the drop down lists and maybe AJAX to request calls. I am not familiar with AJAX but just have some preliminary information. Please suggest of how to get the required information.

Here's the link(thai language). It has all the business information displayed on a map.

https://xn--b3cuh3bhdeppad0as7a5dybu5qd4a3kl8e.moc.go.th/MarkerShop#gomap

Answer 1

try this:

import requests as r
import json
from bs4 import BeautifulSoup

my_url = 'https://xn--b3cuh3bhdeppad0as7a5dybu5qd4a3kl8e.moc.go.th/MarkerShop'

res = r.get(my_url)
soup = BeautifulSoup(res.text, "html.parser")
provinces = soup.find_all('option')[1:]
provinces.pop()
invalid_tags = ['\\r', '\\n', '<', '>', '-', ';','<br>','<b>','</b>','/b',"img src='icon/icon_y.png'"," ","brb","imgsrc='icon/icon_g.png'"]
master_list=[]
for province in provinces:
    res = r.get('https://xn--b3cuh3bhdeppad0as7a5dybu5qd4a3kl8e.moc.go.th/api/shopapi/getshopicon?prov={}&amphur=0'.format(province["value"]))
    data = res.json()
    for detail in data:
        data_dict={}
        for invalid_tag in invalid_tags:
            detail['Address']  = detail['Address'].replace(invalid_tag, '')
            detail['ShopName'] = detail['ShopName'].replace(invalid_tag,'')
        data_dict["Address_new"]=detail['Address']
        data_dict['Shop']=detail['ShopName']
        master_list.append(data_dict)
print(master_list)

This gets all the data for each province with the data of all it's districts, you'll get it as JSON you could extract all your needs from there.