Web scraping the Uber price estimator page using Python

Question

So I'm really new to web scraping and I want to build a bot that checks the Uber ride price from point A to point B over a period of time. I used the Selenium library to input the pickup location and the destination and now I want to scrape the resulting estimated price from the page.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Initialize webdriver object
firefox_webdriver_path = '/usr/local/bin/geckodriver'
webdriver = webdriver.Firefox(executable_path=firefox_webdriver_path)

webdriver.get('https://www.uber.com/global/en/price-estimate/')

time.sleep(3)

# Find the search box
elem = webdriver.find_element_by_name('pickup')
elem.send_keys('name/of/the/pickup/location')

time.sleep(1)
elem.send_keys(Keys.ENTER)

time.sleep(1)
elem2 = webdriver.find_element_by_name('destination')
elem2.send_keys('name/of/the/destination')

time.sleep(1)
elem2.send_keys(Keys.ENTER)
time.sleep(5)

elem3 = webdriver.find_element_by_class_name('bn rw bp nk ih cr vk')
print(elem3.text)

Unfortunately, there's an error:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .bn rw bp nk ih cr vk

And I can't seem to figure out a solution. While inspecting the page, the name of the class that holds the price has the following name "bn rw bp nk ih cr vk" and after some searches I found out this might be Javascript instead of HTML. (I would also point out that I'm not really familiar with them.)

Inspecting the page

Eventually, I thought I could use BeautifulSoup and requests modules and faced another error.

import requests
from bs4 import BeautifulSoup
import re
import json

response = requests.get('https://www.uber.com/global/en/price-estimate/')
print(response.status_code)

406

I also tried changing the User Agent in hope of resolving this HTTP error message, but it did not work. I have no idea how to approach this.

Answer 1

Not exactly what you need. But I recently made a similar application and I will provide part of the function that you need. The only thing you need is to get the latitude and longitude. I used google_places provider for this, but iam sure there is many free services for this.

import requests
import json


def get_ride_price(origin_latitude, origin_longitude, destination_latitude, destination_longitude):
    url = "https://www.uber.com/api/loadFEEstimates?localeCode=en"

    payload = json.dumps({
      "origin": {
        "latitude": origin_latitude,
        "longitude": origin_longitude
      },
      "destination": {
        "latitude": destination_latitude,
        "longitude": destination_longitude
      },
      "locale": "en"
    })
    headers = {
      'content-type': 'application/json',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
      'x-csrf-token': 'x'
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    result = [[x['vehicleViewDisplayName'], x['fareString']] for x in response.json()['data']['prices']]
    return result


print(get_ride_price(51.5072178, -0.1275862, 51.4974948, -0.1356583))

OUTPUT:

[['Assist', '£13.84'], ['Access', '£13.84'], ['Green', '£13.86'], ['UberX', '£14.53'], ['Comfort', '£16.02'], ['UberXL', '£17.18'], ['Uber Pet', '£17.77'], ['Exec', '£20.88'], ['Lux', '£26.32']]