Parse html element with lxml / xpath

Question

Using lxml/python and xpath, I retrieved the value betwen my tags. I would like to get the html properties too, not just the text, my programme works, but skipped two lines.

python :

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import lxml.html
htmltree = lxml.html.parse('data.html')
res = htmltree.xpath("//table[@class='mainTable']/tr/td/text()")
print '\n'.join(res).encode("latin-1")

data.html sample

<table class='mainTable'>
         <TR>
                  <TD bgcolor="#cccccc">235</TD>
                  <TD bgcolor="#cccccc"> Windows XP / Office 2003.</TD>
                  <TD bgcolor="#cccccc">
                  G:\REMI\projets\Migration_XP_Office2003\Procedures\Installation Win XP et Office 2003.doc</TD>
                  <TD bgcolor="#cccccc">2005-10-18</TD>
                  <TD bgcolor="#cccccc">2010-12-30</TD></TR>
                  <TD bgcolor="#cccccc">
                  <P class="MsoBodyText" 
                    style="margin: 0cm 0cm 0pt;"><STRONG><FONT face="Times New Roman" size="5">blablablablablablbala<BR><BR></FONT></STRONG></FONT></P>
                  </TD>
                <TR>
                  <TD bgcolor="#cccccc">23</TD>
                  <TD bgcolor="#cccccc">XEROX/ MAC</TD>
                  <TD bgcolor="#cccccc">
                    <P>joint.</P>
                    <P>&nbsp;</P></TD>
                  <TD bgcolor="#cccccc">G:\DDTH_INF\REMI\bdcfiles\I098_Page_de_garde_MAC.doc</TD>
                  <TD bgcolor="#cccccc">2012-12-19</TD>
                  <TD bgcolor="#cccccc">2012-12-19</TD>
         </TR>
 </table>

return :

 235 Windows XP / Office 2003.
 G:\REMI\projets\Migration_XP_Office2003\Procedures\Installation Win XP
 et Office 2003.doc 2005-10-18 2010-12-30

 23 XEROX/ MAC G:\DDTH_INF\REMI\bdcfiles\I098_Page_de_garde_MAC.doc
 2012-12-19 2012-12-19

I dont understand why the programm skipped

<P class="MsoBodyText" 
                        style="margin: 0cm 0cm 0pt;"><STRONG><FONT face="Times New Roman" size="5">blablablablablablbala<BR><BR></FONT></STRONG></FONT></P>

and

 <P>joint.</P>
                        <P>&nbsp;</P>

Because it's between <p> tag ? I just want to get all the data between each TD. I tried with /tr/td/p/ too but it's not the solution.

note: this code is a sample, its possible that html is broken, but my file is well structured.

Answer 1

This is because you are getting the text() out of each td element - which basically means - give me a text node located directly inside the td element .

Instead, call .text_content() on every td found:

texts = [td.text_content() for td in htmltree.xpath("//table[@class='mainTable']/tr/td")]