How to write a Python script to search for a keyword in a particular website's database using the website's search bar?

Question

I want to search for a keyword in a particular website using its search bar. For example I want to search about "birds" in Wikipedia. For that I have to open Google Chrome, then open Wikipedia, then search for the word "birds" in Wikipidea's search engine.

I want to automate this process using Python. I'm using PyCharm.

Answer 1

If emulating a browser user activity is OK to you, you may consider installing Selenium and Chrome webdriver (here is the instruction: https://pypi.org/project/selenium/ ). "Example 1" is similar to the solution of your problem.

The search bar is <input type="search" name="search" placeholder="Search Wikipedia" title="Search Wikipedia [alt-shift-f]" accesskey="f" id="searchInput" tabindex="1" autocomplete="off"> element and has "searchInput" id, which you can use to select it with el = browser.find_element_by_id("searchInput") Then use el.send_keys('birds' + Keys.RETURN) to fill the input with your request and search.

So the script may look like the following:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Chrome()

browser.get('https://en.wikipedia.org/wiki/Main_Page')

print("Enter a keyword to search on wikipedia: ", end='')
keyword = input()

elem = browser.find_element_by_id('searchInput')  # Find the search box
elem.send_keys(keyword + Keys.RETURN)

# do something with the opened page

browser.quit()

If you don't want to emaluate browser activity, you may resolve it somehow with requests and BeautifulSoup4 modules, but the solution will be more complex, though, probably, more efficient