使用Python从网站检索源代码

Question

I've been trying to extract links from a website with no luck. From what I've read it can be done easily, but the links are with a pop-up dialog within the website. The only way I can grab the links would be to ctrl-A and view the source to copy.

Is there a way to Select all before grabbing the entire content?

Appreciate any information or pointers!

EDIT I would like to avoid downloading anything beyond what python already has, eg BS/Scrapy etc.

Answer 1

As far as retrieving links are concerned, it can be done using requests and bs4 as suggested by jonrsharpe. I am excited in answering this because, i wrote one of them 1 or 2 days ago.

from sys import argv
import requests
from bs4 import BeautifulSoup
#from notify2 import notify2
from time import sleep
import notify2
def send_message(title, message):
    notify2.init("Init")
    notice = notify2.Notification(title, message)
    notice.show()
    return

url = "http://stackoverflow.com/feeds/tag?tagnames=%s&sort=newest" % argv[1]
while True:
    r = requests.get(url)
    while r.status_code is not 200:
            r = requests.get(url)
    soup = BeautifulSoup(r.text)
    data = soup.find_all("link")
    question = data[2].get('href')
    question = question[question.find('questions') + 19:]
    send_message("Question %s: " % argv[1].upper(), question)
    sleep(60)

Basically, this a script which gives you desktop notifications every 1 min. The data which is shown is the first question of specified tag of stackoverflow (in most of the cases it works just fine, you have to check for correct url in others)
Here, you can reach a url and get all the data using requests.get() and parse those data using different methods given by bs4
By the way, the repo of this code is here . Any contributions would be appreciated.