带有python漂亮汤的HTML表

Question

I have a HTML table which looks like this :

<table border=0 cellspacing=1 cellpadding=2 class=form>
<tr class=form><td class=formlabel>Heating Coils in Bunker Tanks</td><td class=form>N</td></tr>
<tr class=forma><td class=formlabel>Heating Coils in Cargo Tanks</td><td class=form>U</td></tr>
<tr class=form><td class=formlabel>Manifold Type</td><td class=form>N</td></tr>
<tr class=forma><td class=formlabel>No. Holds</td><td class=form>5</td></tr>
<tr class=form><td class=formlabel>No. Centreline Hatches</td><td class=form>5</td></tr>
<tr class=forma><td class=formlabel>Lifting Gear</td><td class=form>Yes</td></tr>
<tr class=form><td class=formlabel>Gear</td><td class=form>4 Crane (30.5t SWL)</td></tr>
<tr class=forma><td class=formlabel>Alteration</td><td class=form>Unknown</td></tr>
</table>

I am using Beautiful soup to extract specific data which comes as a response from a scrapy spider

soup = BeautifulSoup(response.body_as_unicode())
table= soup.find('table', {'class': 'form'})
# psusedo code find manifold type and number of Holds

How do i go about doing this.Do note that the ordering of the values might change but the form label always remains the same ? How do i search using a specific form label ?

Edit:

<tr class=forma><td class=formlabel>Fleet Manager (Operator)</td><td class=form><a href="oBasic.asp?LRNumber=9442964&Action=Display&LRCompanyNumber=40916">ESSAR SHIPPING LTD</a></td></tr>

this particular case scenario does not work with the following sibling search ? How to overcome this ?

Answer 1

You can find the td element by text and get the next sibling :

table.find('td', text='Manifold Type').next_sibling.text

As a side note, why do you need to use BeautifulSoup inside a Scrapy spider? Scrapy itself is pretty powerful in terms of HTML parsing, locating elements:

response.xpath('//table[@class="form"]//td[.="Manifold Type"]/following-sibling::td/text()')

Demo from the scrapy shell :

$ scrapy shell index.html
In [1]: response.xpath('//table[@class="form"]//td[.="Manifold Type"]/following-sibling::td/text()').extract()
Out[1]: [u'N']