使用 Selenium & Beautiful Soup 从网站数据中获取动态表格

Question

from selenium import webdriver
from bs4 import BeautifulSoup
path = 'C:/Program Files/Google/Chrome/Application/ChromeDriver'
driver = webdriver.Chrome(executable_path = path)
driver.get("https://www1.nseindia.com/products/content/equities/equities/oi_spurts.htm")
soup = BeautifulSoup(driver.page_source,"lxml")
for items in soup.select('#profile table.table tr'):
    data = [item.get_text(strip=True) for item in items.select("th,td")]
    print(data)

I am trying to get a dynamic tabular data from website using selenium and chrome driver. Need some help in getting data as right now i am getting empty dataframe.

Answer 1

You're one request away from getting that table. It comes in as a plain JSON so all you need is requests .

For example:

import requests
from tabulate import tabulate

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0",
    "X-Requested-With": "XMLHttpRequest",
}
url = "https://www1.nseindia.com/live_market/dynaContent/live_analysis/oi_spurts/topPositiveOIChangeData.json"
response = requests.get(url, headers=headers).json()
print(tabulate(response["data"], headers="keys", showindex=False, tablefmt="pretty"))

Output:

+------------+-----------+-----------+----------+--------------+-------------+-----------------+------------+------+-----------+-------------+--------------+----------+
|   symbol   | latestOI  |  prevOI   | oiChange | percOIchange |   volume    |  valueInLakhs   | underlying | isFO |  FUTVAL   |   OPTVAL    |    TOTVAL    |  OPVAL   |
+------------+-----------+-----------+----------+--------------+-------------+-----------------+------------+------+-----------+-------------+--------------+----------+
| JINDALSTEL |   9,098   |   6,211   |  2,887   |    46.48     |   16,723    |   2,00,623.71   |   235.05   |  0   | 1,111.20  |   895.00    |   2,006.20   |  19.53   |
|  INFRATEL  |   9,707   |   6,987   |  2,720   |    38.93     |   57,630    |   3,40,222.56   |   221.70   |  0   | 1,111.91  |  2,290.29   |   3,402.20   |  65.60   |
|    GAIL    |   9,270   |   6,793   |  2,477   |    36.46     |   22,653    |   1,37,091.48   |   97.60    |  0   |  430.14   |   940.80    |   1,370.94   |  20.17   |
| BAJAJFINSV |  24,693   |  18,452   |  6,241   |    33.82     |  1,01,363   |  10,56,613.81   |  8,548.80  |  0   | 3,050.60  |  7,515.50   |  10,566.10   |  136.56  |
| BANKNIFTY  | 8,99,992  | 6,73,712  | 2,26,280 |    33.59     | 1,37,75,584 | 10,04,67,151.05 | 29,236.00  |  1   | 30,772.13 | 9,73,899.33 | 10,04,671.46 | 8,435.93 |
| ADANIPORTS |  34,066   |  26,191   |  7,875   |    30.07     |   39,050    |   3,72,251.30   |   375.95   |  0   | 2,629.21  |  1,093.32   |   3,722.53   |  14.28   |
|   TITAN    |  16,910   |  13,234   |  3,676   |    27.78     |   76,457    |   7,83,224.88   |  1,362.05  |  0   | 1,876.69  |  5,955.54   |   7,832.23   |  88.77   |
| AMARAJABAT |   2,376   |   1,913   |   463    |    24.20     |    8,709    |    76,247.76    |   862.00   |  0   |  205.81   |   556.65    |    762.46    |   6.49   |
|   INDIGO   |  11,782   |   9,593   |  2,189   |    22.82     |   38,356    |   3,23,486.48   |  1,675.35  |  0   | 1,419.05  |  1,815.81   |   3,234.86   |  31.15   |
| TATASTEEL  |  44,455   |  36,295   |  8,160   |    22.48     |   84,064    |   7,67,276.96   |   531.55   |  0   | 3,171.72  |  4,501.05   |   7,672.77   |  87.21   |
| TATAPOWER  |  11,705   |   9,598   |  2,107   |    21.95     |    7,330    |    58,594.30    |   58.10    |  0   |  365.53   |   220.41    |    585.94    |   3.56   |
| AUROPHARMA |  34,318   |  28,559   |  5,759   |    20.17     |   26,876    |   1,47,356.23   |   833.05   |  0   |  800.34   |   673.24    |   1,473.58   |  10.00   |
|    HDFC    | 1,27,756  | 1,06,830  |  20,926  |    19.59     |   77,486    |   5,51,136.78   |  2,329.00  |  0   | 2,666.28  |  2,845.07   |   5,511.35   |  31.63   |
and so on ..