Scraping Website Needing Interaction

Question

I'm working on a scraping project - looking at what recylcing companies offer for different products in the UK

I've run into a problem with this website:

http://www.musicmagpie.co.uk/entertainment/

I have a list of barcodes I want to find their buy price for (enter barcode into search box and hit 'Add button). I've managed to get a Selenium Webdriver working, but it's a very slow process and I can't run through lots of barcodes without the website crapping out at me and killing my process at some point.

I'm aiming for about 1 search per sec, at the moment it's taking be about 5+ secs on average. This is the code I'm running:

driver = webdriver.Chrome(r"C:\Users\leonK\Documents\Python Scripts\chromedriver.exe")
driver.get('http://www.musicmagpie.co.uk/start-selling/basket-media')

countx = 0
count = 0
for EAN in EANs:
    countx += 1
    count += 1

    if count % 200 == 0:
        driver.close()
        driver = webdriver.Chrome(r"C:\Users\leonK\Documents\Python Scripts\chromedriver.exe")
        driver.get('http://www.musicmagpie.co.uk/start-selling/basket-media')
        count = 1

    driver.find_element_by_xpath("""//*[@id="txtBarcode"]""").send_keys(str(EAN))

    #If popup window appears, exception will close it as first click will fail.
    try:    
        driver.find_element_by_xpath("""//*[@id="getValSmall"]""").click()
    except:
        driver.find_element_by_xpath("""//*[@id="gform_close"]""").click()

    prodnames = driver.find_elements_by_xpath("""//div[@class='col_Title']""")
    if len(prodnames) == count:
        ProductName.append(prodnames[0].text)
        BuyPrice.append(driver.find_elements_by_xpath("""//div[@class='col_Price']""")[0].text)
    else:
        ProductName.append('nan')
        BuyPrice.append('nan')
        count = len(prodnames)

    elapsed = time.clock()    
    print('MusicMagpieScraper:', EAN, '--', countx, '/', len(EANs), '--', (elapsed - start), 's')

driver.close()

I've got some experience using Urllib and parsing with BeautifulSoup, and would prefer to switch over to that. But, I don't know how to extract that data without the webdriver doing the clicks.

Any advice/tips would be very appriciated!

Added:

The add button link is:

__doPostBack('ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$getValSmall','')

This is the JS function I found:

{name: "__EVENTTARGET", value: ""}
{name: "__EVENTARGUMENT", value: ""}
{name: "__VIEWSTATE", value: "/wEPDwUENTM4MQ9kFgJmD2QWAmYPZBYCZg9kFgJmD2QWBGYPZB…uZSAhaW1wb3J0YW50O2RkQweS+jvDtjK8er7dCKBBRwOWWuE="}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$signIn_8$hdn_BasketValue", value: "2"}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$txtBarcode", value: "5051275026429"}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$wtmBarcode_ClientState", value: ""}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedTechVal_11$txtSearch", value: "Enter item (e.g. iPhone 5)"}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedTechVal_11$wmSearch_ClientState", value: ""}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$LegoVal_12$ddlLego", value: "-999"}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$TotalValueBox_14$txtPromoVoucher_sm", value: ""}
{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$TotalValueBox_14$txtPromoVoucher", value: ""}
{name: "__SCROLLPOSITIONX", value: "0"}
{name: "__SCROLLPOSITIONY", value: "0"}
{name: "hiddenInputToUpdateATBuffer_CommonToolkitScripts", value: "1"}

line 4 is where the barcode is input:

{name: "ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$txtBarcode", value: "5051275026429"}

Hopefully useful info, I don't know where to go from here and google hasn't helped too much

Answer 1

I managed to find a solution to this using requests

    get_response = requests.get(url='http://www.musicmagpie.co.uk/start-selling/')
    post_data = {'__EVENTTARGET' : 'ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$getValSmall',
           '__EVENTARGUMENT' : '',
           'ctl00$ctl00$ctl00$ContentPlaceHolderDefault$mainContent$tabbedMediaVal_10$txtBarcode' : ean}
    # POST some form-encoded data:
    post_response = requests.post(url='http://www.musicmagpie.co.uk/start-selling/', data=post_data)    

    soup = BeautifulSoup(post_response.text, "lxml")

    BuyPrice = soup.find('div', class_='col_Price').text.rstrip()
    ProductName = soup.find('div', class_='col_Title').text.rstrip()

This code sends a dictionary of functions/values (may not be correct terminology!) and it fires back an easy-to-parse response from which I pulled the data I wanted!