谷歌地图使用硒的地方ID

Question

from selenium import webdriver
import re
driver= webdriver.Chrome(executable_path=r"C:\Users\chromedriver")
sentence = "chiropractor in maryland"
url="https://google.com/search?hl=en&q={}".format(sentence)
driver.get(url)
links=driver.find_elements_by_xpath('//a[@href]')
maps=[i for i in links if i.text=="Maps"][0].click()
html=driver.page_source
#ChIJaYGxdRj9t4kRcJmJlvQkKX0
#ChIJCf4MzWjgt4kRluBnhQTHlBM
#ChIJBXxr8brIt4kRVE-gIYDyV8c
#ChIJX0W_Xo4syIkRUAtRFy8nz1Y place ids in html

Hello, this is my first selenium project I am trying to find the places ids from result I have added some of place id (i got using API), I tried to find them in inspector tools but I couldn't,however, they are available in the page source I tried using regex it seems that they follow the following path

2,[null,null,\\"bizbuilder:gmb_web\\",[6,7,4,1,3]\\n]\\n]\\n]\\n,1,null,null,null,null,null,null,[\\"-8523065488279764631\\",\\"9018780361702349168\\"]\\n]\\n]\\n]\\n,null,null,null,[[\\"chiropractor\\"]\\n]\\n,null,\\"ChIJaYGxdRj9t4kRcJmJlvQkKX0\\",null,null,null,[\\"South Gate\\",\\"806 Landmark Dr Suite 126\\",\\"806 Landmark Dr Suite 126\\",\\"Glen Burnie\\"]\\n,null,null,null,null,null,[null,\\"SearchResult.TYPE_PERSONAL_

after "\\"chiropractor\\"]\\n]\\n,null,\\" Place ID ",null ...

but I can't find the regex for it. I need help writing the correct regex or find another way of finding palce_id. I hope that no one answers with refer to using their API

Answer 1

I think this could be improved but the string itself sits in a script tag that has window.APP_OPTIONS in it. Each of those ids starts with ChIJ , has a defined character set following and is of length 27 in total.

I have also started directly with the map page rather than click to it. I didn't need a wait condition despite several runs. This could be added if wanted/required.

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import re

sentence = "chiropractor in maryland"
url = 'https://www.google.com/maps/search/{}'.format(sentence)
d = webdriver.Chrome()
d.get(url)
soup = bs(d.page_source, 'lxml')

for script in soup.select('script'):
    if 'window.APP_OPTIONS' in script.text:
        script = script.text
        break    
r = re.compile(r'(ChIJ[a-zA-Z\.0-9\-\_]{23})')
items = r.findall(script)
print(items)

d.quit()

---

A little riskier you could work off page_source direct

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import re

sentence = "chiropractor in maryland"
url = 'https://www.google.com/maps/search/{}'.format(sentence)
d = webdriver.Chrome()
d.get(url)
r = re.compile(r'(ChIJ[a-zA-Z\.0-9\-\_]{23})')
items = r.findall(d.page_source)
print(items)

d.quit()

---

Notes:

I am specifying a pattern designed to only match the required items currently (for given search). It is conceivable, in future/new searches, that pattern could occur and not be an id. The page_source is a larger search space and therefore a greater likelihood of encountering an unwanted string that matches the pattern. The script tag is not only where you would expect to find the ids but is also a smaller search space. Over time you might also want to check character set does not require any additional characters for matching new ids. You can easily check against the result per page count.