當我用 python 刮擦 web 時出現 403 錯誤

Question

import requests
from bs4 import BeautifulSoup
response = requests.get(
'https://xueqiu.com/hq#exchange=HK&firstName=2&secondName=2_0')
soup = BeautifulSoup(response.text, "html.parser")
print(soup.prettify()) 

因 403 Forbidden 而失敗 我試圖從表中獲取庫存數據但失敗了

Answer 1

從頁面集User-Agent header 中獲取數據。 此外，數據從外部 URL 加載了 JavaScript，因此請改用此 URL：

import requests
import pandas as pd

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0"
}

api_url = "https://xueqiu.com/service/v5/stock/screener/quote/list"

params = {
    "page": "1",
    "size": "30",
    "order": "desc",
    "orderby": "percent",
    "order_by": "percent",
    "market": "HK",
    "type": "hk",
    "is_delay": "true",
}

data = requests.get(api_url, params=params, headers=headers).json()

df = pd.DataFrame(data["data"]["list"])
print(df)

印刷：

   symbol  net_profit_cagr north_net_inflow       ps  type  percent  has_follow  tick_size pb_ttm  float_shares  current  amplitude       pcf  current_year_percent  float_market_capital north_net_inflow_time  market_capital  dividend_yield  lot_size     roe_ttm  total_percent  percent5m  income_cagr        amount    chg  issue_date_ts     eps  main_net_inflows    volume  volume_ratio       pb  followers  turnover_rate  first_percent            name    pe_ttm  total_shares limitup_days
0   08070              NaN             None   0.2087    30    62.71       False      0.005   None     200000000    0.480      76.27       NaN                128.57              96000000                  None        96000000             NaN      5000   15.349430         -83.28      62.71    27.336445  1.782500e+04  0.185  1456243200000  0.0482               0.0     55000          0.78   1.3801        123           0.03         -56.10          僑洋國際控股    9.9678     200000000         None
1   00593              NaN             None   5.0176    30    42.22       False      0.010   None     285490845    1.280      38.89       NaN                 58.02             365428282                  None       365428282             NaN       500 -255.676498         104.67       0.00   -31.574156  1.225000e+03  0.380   743702400000 -3.0146               0.0      1000          0.76      NaN        775           0.00            NaN             夢東方       NaN     285490845         None
2   02246        -6.590145             None   3.8489    30    37.15       False      0.010   None     615527987    6.940      36.36       NaN                -67.72            4271764230                  None      4271764230             NaN       200   47.736094         -67.72       1.31    11.935704  1.021789e+08  1.880  1656000000000 -1.7344          964130.0  16396666          7.31      NaN       1009           2.66         -22.23            快狗打車       NaN     615527987         None
3   01626              NaN             None   0.2738    30    33.77       False      0.010   None     300000000    1.030     112.99    5.5130                -45.79             309000000                  None       309000000             NaN      2000  -15.133485          -7.16      19.77     6.700910  1.384240e+06  0.260  1403798400000 -0.1069          327240.0   1332000          4.74   1.5749        398           0.44           0.00            嘉耀控股       NaN     300000000         None
4   08287        56.499210             None   0.0871    30    31.91       False      0.001   None     990000000    0.062      57.45       NaN                -16.22              61380000                  None        61380000             NaN     10000  -37.909770         -86.67       0.00   -17.118113  1.164000e+05  0.015  1516204800000 -0.0535               0.0   1710000          5.94   0.4917        422           0.17         -19.35            百家淘客       NaN     990000000         None
5   08143       -36.837585             None   2.4941    30    31.25       False      0.001   None    2818249944    0.063      50.00       NaN                  8.62             177549746                  None       177549746             NaN      4000  -36.347751            NaN       0.00   -45.052062  1.072880e+05  0.015  1020960000000 -0.0045               0.0   2092000          2.57   6.0001       3156           0.07            NaN            金威醫療       NaN    2818249944         None
6   08217        49.348042             None   0.3905    30    30.00       False      0.001   None     301860000    0.156      28.33       NaN                -34.18              47090160                  None        47090160             NaN     10000  -47.252941         -99.45       0.00   -40.158691  3.800812e+04  0.036  1460390400000 -0.0667               0.0    268750         14.04   0.7782        615           0.09         105.00            聯旺集團       NaN     301860000         None
7   08588              NaN             None      NaN    30    30.00       False      0.001   None     417503991    0.013     120.00       NaN                 30.00               5427552                  None         5427552             NaN      5000   12.552176         -35.00       0.00    -1.904225  3.180092e+05  0.003  1659974400000  0.0927               0.0  21491400          1.55   0.0154         14           5.15          30.00        樹熊金融集團股權    0.1402     417503991         None
8   02339              NaN             None   0.1391    30    26.98       False      0.010   None     574339068    0.800      34.92       NaN                  5.26             459471254                  None       459471254             NaN      2000   -1.016585            NaN      -1.23    -8.694416  5.841924e+06  0.170  1065715200000 -0.0162          763940.0   7250400        100.78   0.5123       1873           1.26            NaN            京西國際       NaN     574339068         None
9   02237              NaN             None   3.7547    30    24.56       False      0.010   None    1600000000    0.710      28.07    9.8390                118.46            1136000000                  None      1136000000             NaN     10000   30.413122          61.36       1.43          NaN  3.650983e+07  0.140  1658073600000  0.0408         2900778.5  56736150          5.78   3.0961        456           3.55         143.08            中國石墨   17.4175    1600000000         None
10  08483        49.536488             None   2.8314    30    20.55       False      0.001   None     800000000    0.088      13.70       NaN                -31.78              70400000                  None        70400000             NaN      8000  -24.539082         -70.25       0.00   -31.092784  2.080000e+03  0.015  1519747200000 -0.0118               0.0     24000          0.20   2.0239        259           0.00           0.00            名仕快相       NaN     800000000         None

...