网页抓取：HTTPError：HTTP 错误 403：禁止，python3

Question

Hi I am need to scrape web page end extract data-id use Regular expression

Here is my code :

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("https://clarity-project.info/tenders/?entity=38163425&offset=100")
bsObj = BeautifulSoup(html,"html.parser")
DataId = bsObg.findAll("data-id", {"skr":re.compile("data-id=[0-9,a-f]")})
for DataId in DataId:
    print(DataId["skr"])

when I run my program in Jupyter :

HTTPError: HTTP Error 403: Forbidden

Answer 1

It looks like the web server is asking you to authenticate before serving content to Python's urllib. However, they serve everything neatly to wget and curl and https://clarity-project.info/robots.txt doesn't seem to exist, so I reckon scraping as such is fine with them. Still, it might be a good idea to ask them first.

As for the code, simply changing the User Agent string to something they like better seems to work:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from urllib.request import urlopen, Request

request = Request(
    'https://clarity-project.info/tenders/?entity=38163425&offset=100',
    headers={
        'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0'})

html = urlopen(request).read().decode()

(unrelated, you have another mistake in your code: bsObj ≠ bsObg)

EDIT added code below to answer additional question from the comments:

What you seem to need is to find the value of the data-id attribute, no matter to which tag it belongs. The code below does just that:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

url = 'https://clarity-project.info/tenders/?entity=38163425&offset=100'
agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36\
(KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'

request = Request(url, headers={'User-Agent': agent})

html = urlopen(request).read().decode()

soup = BeautifulSoup(html, 'html.parser')

tags = soup.findAll(lambda tag: tag.get('data-id', None) is not None)
for tag in tags:
    print(tag['data-id'])

The key is to simply use a lambda expression as the parameter to the findAll function of BeautifulSoup.

Answer 2

The server is likely blocking your requests because of the default user agent. You can change this so that you will appear to the server to be a web browser. For example, a Chrome User-Agent is:

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 

To add a User-Agent you can create a request object with the url as a parameter and the User-Agent passed in a dictionary as the keyword argument 'headers'.

See:

import urllib.request
r = urllib.request.Request(url, headers= {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})
html = urllib.request.urlopen(r)

Answer 3

You could try with this:

#!/usr/bin/env python

from bs4 import BeautifulSoup
import requests 

url = 'your url here'
soup = BeautifulSoup(requests.get(url).text,"html.parser")

for i in soup.find_all('tr', attrs={'class':'table-row'}):
    print '[Data id] => {}'.format(i.get('data-id'))

This should work!