如何在将pyquery对象转换为字符串时取消特殊字符

Question

I am trying to fetch a remote page with python requests module, reconstruct a DOM tree, do some processing and save the result to file. When I fetch a page and then just write it to the file everything works (I can open an html file later in the browser and it is rendered correctly).

However, if I create a pyquery object and do some processing and then save it by using str conversion it fails. Specifically, special characters like && and etc. get modified within script tags of the saved source (caused by application of pyquery) and it prevents page from rendering correctly.

Here is my code:

import requests
from lxml import etree
from pyquery import PyQuery as pq

user_agent = {'User-agent': 'Mozilla/5.0'}
r = requests.get('http://www.google.com',headers=user_agent, timeout=4)

DOM = pq(r.text)
#some optional processing
fTest = open("fTest.html","wb")
fTest.write(str(DOM))
fTest.close()

So, the question is: How to make sure that special characters aren't escaped after application of pyquery? I suppose it might be related to lxml (parent library for pyquery), but after tedious search online and experiments with different ways of object serialization I still didn't make it. Maybe this is also related to unicode handling?!

Many thanks in advance!

Answer 1

I have found an elegant solution to the problem and the reason why it the code didn't work before.

First, you can read carefully the page with http://lxml.de/lxmlhtml.html . It has a section "Creating HTML with the E-factory" . After the section they point out to the fact that etree.tostring() method works for XML only. But for HTML with additional possibility to have script or style tags it will mess things around. So.. Second, the solution is to use the overloaded method html.tostring() .

The final working code is:

# for networking
import requests
# for parsing and serialization
from lxml import etree
from lxml.html import tostring as html2str # IMPORTANT!!!
from pyquery import PyQuery as pq

user_agent = {'User-agent': 'Mozilla/5.0'}
r = requests.get('http://www.google.com',headers=user_agent, timeout=4)

# construct DOM object
DOM = pq(r.text)
# do stuff with DOM
#
# save result to file
fTest = open("fTest.html","wb")
fTest.write(html2str(DOM.root)) # IMPORTANT!!!
fTest.close()

Hope it will save time some of you in future! Have fun guys! ;)