如何使用 lxml 从 html 锚点中提取 href url？

Question

I try to extract the next page href string using lxml.

For example I try to extract the "/review/bulb.co.uk?b=MTYxOTg5MDE1OTAwMHw2MDhkOGZlZmY5ZjQ4NzA4ZTA4MWI2Mzk" from the html in the following example:

<nav rel="nav" class="pagination-container AjaxPager">
    <a href="/review/bulb.co.uk?b=MTYxOTg5MDE1OTAwMHw2MDhkOGZlZmY5ZjQ4NzA4ZTA4MWI2Mzk" data-page-number="next-page" class="button button--primary next-page" rel="next" data-track-link="{'target': 'Company profile', 'name': 'navigation', 'navigationType': 'next'}">
Next page
    </a>
</nav>

I have tried the following but it returns a list not the string that I am looking for:

import requests
import lxml.html as html

URL = https://uk.trustpilot.com/review/bulb.co.uk
page = requests.get(URL)

tree = html.fromstring(page.content)

href = tree.xpath('//a/@href')

Any idea what I am doing wrong?

Answer 1

Making this change to your code

href = tree.xpath('//a[@class="button button--primary next-page"]/@href')
href[0]

Gives me this output:

'/review/bulb.co.uk?b=MTYxOTk1ODMxMzAwMHw2MDhlOWEyOWY5ZjQ4NzA4ZTA4MjMxNTE'

which is close to the output in your question (its value may change dynamically).