使用LXML解析HTML数据

Question

<div id="descriptionmodule" class="module toggle-wrap">
    <div class="mod-header">
        <h3 class="toggle-title">Description</h3>
    </div>
    <div id="issue-description" class="mod-content">
        <p>qqqqqqqqqqqqq,<br/>
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq<br/>
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.</p>

<p>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq</p>

<p>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.</p>

<ul class="alternate" type="square">
    <li>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq</li>
</ul>

I want only the Q's . I tried this

doc=lh.fromstring(resp.read())  
for id in doc.cssselect('div.mod-content' ):
    print id.text_content()

This gives me the q's but it also gives me other details on the page with class mod-content. How do i specifically get only the q's.

I am using lxml.

<div id="peoplemodule" class="module toggle-wrap">
    <div class="mod-header">
        <h3 class="toggle-title">People</h3>
    </div>
    <div class="mod-content">
        <ul class="item-details" id="peopledetails">
            <li class="people-details">
                                <dl>
                    <dt>Assignee:</dt>
                    <dd id="Assign-Val">
                                <a class="user-hover" rel="605794069" id="issue_summary_assignee_605794069" href="--------------"> AAAAAAAAAAAAA a>
                    </dd>
                </dl>
                                                <dl>
                    <dt>Reporter:</dt>
                    <dd id="Report-Val">
                                <a class="user-hover" rel="700843051" id="issue_summary_reporter_700843051" href="-------------------------">BBBBBBBBBBBBBB</a>
                    </dd>
                </dl>
                                <dl><dt>&nbsp;</dt><dd>&nbsp;</dd></dl>
                                <dl>
                    <dt title="Multiple Assignees">Multiple Assignees:</dt>
                    <dd id="customfield_10020-val">    <div class="shorten" id="customfield_10020-field">
                                    <span class="tinylink">        <a class="user-hover" rel="604810609" id="multiuser_cf_604810609" href------------------">FFFFFFFFFFFFFF</a></span>,                                                 <span class="tinylink">        <a class="user-hover" rel="600548483" id="multiuser_cf_600548483" href="------------------------------------">EEEEEEEEEEEEEEEEE</a></span>                        </div>
</dd>
                </dl>
                            </li>
        </ul>
                        <div id="watchers-val">
                                                <a href="----------------------------------------" id="watching-toggle" rel="858270" title="Start watching this story"><span class="icon icon-watch-off"></span><span class="action-text">Watch</span></a>


                            (<span id="watcher-data">1</span>)
                    </div>
            </div>
</div>

Answer 1

First off: if you are parsing HTML there is a high chance humans will have messed up with it and it won't validate correctly. For example this is the case for the example you posted (there are a couple of </div> missing...). Consider passing to beautifulsoup instead, which is specifically designed to accommodate for these kind of errors.

That said, if your question is just about how to extract the "textual part of the HTML", or in other words how to convert HTML → plain text [as opposed to "extracting only the text contained in specific HTML containers], this is a minimal working example:

from lxml import etree

content = '''<div id="descriptionmodule" class="module toggle-wrap">
    <div class="mod-header">
        <h3 class="toggle-title">Description</h3>
    </div>
    <div id="issue-description" class="mod-content">
        <p>qqqqqqqqqqqqq,<br/>
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq<br/>
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.</p>

<p>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq</p>

<p>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.</p>

<ul class="alternate" type="square">
    <li>qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq</li>
</ul></div></div>'''

tree = etree.fromstring(content)

for bit in tree.xpath('//text()'):
    if bit.strip():  # you can insert any kind of test here
        print bit

It outputs:

Description
qqqqqqqqqqqqq,

qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq.
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

HTH!