XPath 总是返回空列表

Question

I am trying to extract the time value from this website .

Below is the code that I am using

import requests
from lxml import html

page = requests.get('https://beta.nseindia.com/get-quotes/derivatives?symbol=NIFTY&identifier=OPTIDXNIFTY26-12-2019CE12300.00')
tree = html.fromstring(page.content)
test1 = tree.xpath('//*[@id="equity-derivative-op-timeStamp"]/text()')

print(test1)

Result:

[]

How can I get the timestamp value in the "Option Chain" tab of the above page and at that particular xpath?

Answer 1

You're getting back an empty result because if you examine the page source of the URL you're fetching, the equity-derivative-op-timeStamp timestamp is empty:

<span id="equity-derivative-op-timeStamp" class="asondate"></span>

That data is populated via Javascript after the page loads. You won't be able to fetch it using the requests module; you'll need to use something like selenium that drives a real browser capable of processing javascript.

Answer 2

As larsks says in his answer

That data is populated via Javascript after the page loads.

But the data is loaded as XHRs. In Firefox right click on the page, select Inspect Element select Network, select XHR, refresh the page right click on the request of interest and open it in a new tab.

Doing this I have identified that the pagehttps://beta.nseindia.com/api/option-chain-indices?symbol=NIFTY may be of interest to you. It is a JSON file. you can use it like any JSON object:

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0'
}

params = (
    ('symbol', 'NIFTY'),
)

response = requests.get('https://beta.nseindia.com/api/option-chain-indices', headers=headers, params=params)
j = response.json()
print(j['records']['timestamp'])

Outputs:

23-Dec-2019 15:30:00

You need to supply a header for this particular request as above. To determine what headers are needed for a particular web-page in Firefox right click on the page, select Inspect Element select Network refresh the page right click on the request you want select Copy then Copy as cURL paste what you copied into https://curl.trillworks.com then use the generated code, if it works remove headers one at a time until you get a minimal set that works. In Chrome it is a similar process.