具有中间元素的XPath for LXML

Question

I'm trying to scrape some pages with python and LXML. My test page is http://www.sarpy.com/oldterra/prop/PDisplay3.asp?ParamValue1=010558233

I'm having good luck with most of the XPaths. For example,

tree.xpath('/html/body/table/tr[1]/td[contains(text(), "Sales Information")]/../../tr[3]/td[1]/text()')

successfully gets me the date of the first sale listed. I have several other pieces too. However, I cannot get the B&P listed under the sale date. For example the B&P of the first sale is 200639333.

I notice in the page structure that there is a form element preceding the tr of the B&P item. Since it's the next table row, I tried incrementing the tr index as follows:

tree.xpath('/html/body/table/tr[1]/td[contains(text(), "Sales Information")]/../../tr[4]/td[1]/text()')

That returns:

['\r\n           ']

Because of the line breaks and sub element of br and input within the field, I tried making text() into text()[1], text()[2], etc., but no luck.

I tried to base the path off of the adjacent form like this:

tree.xpath('/html/body/table[7]/form[@action="../rod/ImageDisplay.asp"]/following-sibling::tr/td[1]/text()')

No luck.

I figure there are two potential issues: the intermediary form elements that may be breaking the indexing patterns, and the whitespace. I'd appreciate any help in correcting this xpath.

Answer 1

The <tr> you are looking for is the child of the <form> , not its sibling , try -

tree.xpath('/html/body/table/tr[1]/td[contains(text(), "Sales Information")]/../../form[1]/td[1]/text()')

This may get you 200639333 with a lot of whitespaces.

Or -

tree.xpath('/html/body/table[7]/form[@action="../rod/ImageDisplay.asp"]/tr[1]/td[1]/text()')

For all such elements.