Using Beautifulsoup and Selenium to get links from a webpage with certain words

Question

I wrote this code to log in to my FB account and get all grouplinks on the page using Selenuim and BeautifulSoup but the BeautifulSoup usage isn't working correctly.

I want to know how to use Selenuim and BeautifulSoup in the same code.

I don't want to use the Facebook API; I want to use Selenium and BeautifulSoup.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import httplib2
from BeautifulSoup import BeautifulSoup, SoupStrainer


usr = raw_input('--> ')
pwd = raw_input('--> ')
poo = raw_input('--> ')

driver = webdriver.Firefox()
# or you can use Chrome(executable_path="/usr/bin/chromedriver")
driver.get("https://www.facebook.com/groups/?category=membership")
assert "Facebook" in driver.title
elem = driver.find_element_by_id("email")
elem.send_keys(usr)
elem = driver.find_element_by_id("pass")
elem.send_keys(pwd)
elem.send_keys(Keys.RETURN)

scheight = .1
while scheight < 9.9:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight/%s);" % scheight)
    scheight += .01
soup = BeautifulSoup(html)
http = httplib2.Http()
status, response = ('https://www.facebook.com/groups/?category=membership')

count = 0
for link in BeautifulSoup(response, parseOnlyThese=SoupStrainer('a')):
     count = count + 1
print 'Count: ', count 

for tag in BeautifulSoup(('a')):
    if link.has_key('href'):
        if '/groups/' in link['href']:

          print link['href']


elem = driver.find_element_by_css_selector(".input.textInput")
elem.send_keys(poo)
elem = driver.find_element_by_css_selector(".selected")
elem.send_keys(Keys.RETURN)
elem.click()
time.sleep(5)

Answer 1

You never declared html .

Selenium's webdriver has a page_source method that you can use:

soup = BeautifulSoup(driver.page_source)

Update for second error

Your line,

status, response = ('https://www.facebook.com/groups/?category=membership')

is trying to assign that string to both status and response . There's nothing to assign to response , and therefore that variable is undefined.

Answer 2

I think BeautifulSoup isn't returning the links right?

I do find links in BeautifulSoup with

soup = BeautifulSoup( html )
for i in soup.find_all( 'a' ):
if '/groups/' in i.get( 'href' ):
    print( i.get( 'href' ) )