Python Mechanize having trouble with website html

Question

I am trying to use the python module mechanize to fill in the form on a webpage and then download the resultant html. The website is the following:

"xxx"

But first, I just want to list the forms. My code is as follows:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)   # ignore robots
br.set_handle_refresh(False)  # can sometimes hang without this

url = "xxx"
response = br.open(url)
a = response.read()      # the text of the page

for form in br.forms():
    print "Form name:", form.name
    print form

The error list given is quite large:

In [75]: % run -i form_filler.py
---------------------------------------------------------------------------
ParseError                                Traceback (most recent call last)
/home/blake/Python/form_filler.py in <module>()
     19 
     20 
---> 21 for form in br.forms():
     22         print "Form name:", form.name
     23         print form

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_mechanize.pyc in forms(self)
    418         if not self.viewing_html():
    419             raise BrowserStateError("not viewing HTML")
--> 420         return self._factory.forms()
    421 
    422     def global_form(self):

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_html.pyc in forms(self)
    555             try:
    556                 self._forms_genf = CachingGeneratorFunction(
--> 557                     self._forms_factory.forms())
    558             except:  # XXXX define exception!
    559                 self.set_response(self._response)

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_html.pyc in forms(self)
    235             _urljoin=_rfc3986.urljoin,
    236             _urlparse=_rfc3986.urlsplit,
--> 237             _urlunparse=_rfc3986.urlunsplit,
    238             )
    239         self.global_form = forms[0]

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in ParseResponseEx(response, select_default, form_parser_class, request_class, entitydefs, encoding, _urljoin, _urlparse, _urlunparse)
    842                         _urljoin=_urljoin,
    843                         _urlparse=_urlparse,
--> 844                         _urlunparse=_urlunparse,
    845                         )
    846 

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in _ParseFileEx(file, base_uri, select_default, ignore_errors, form_parser_class, request_class, entitydefs, backwards_compat, encoding, _urljoin, _urlparse, _urlunparse)
    979         data = file.read(CHUNK)
    980         try:
--> 981             fp.feed(data)
    982         except ParseError, e:
    983             e.base_uri = base_uri

/usr/local/lib/python2.7/dist-packages/mechanize-0.2.5-py2.7.egg/mechanize/_form.pyc in feed(self, data)
    758             _sgmllib_copy.SGMLParser.feed(self, data)
    759         except _sgmllib_copy.SGMLParseError, exc:
--> 760             raise ParseError(exc)
    761 
    762     def close(self):

ParseError: expected name token at '<!!--end footer-->\r\n'

My understanding is that the html is 'badly written' somehow. I've tried the above code on sample websites such as google, and it runs fine. I suspect that the carriage return has something to do with it, but how do I bypass this problem?

Answer 1

Apparently if I use

br = mechanize.Browser(factory=mechanize.RobustFactory()) 

instead of

br = mechanize.Browser()

It gives the correct output:

In [18]: % run -i form_filler.py
Form name: qSearchForm
<qSearchForm GET xxx
  <TextControl(q=Search)>
  <SubmitControl(qSearchBtn=) (readonly)>>
Form name: form1
<form1 POST xxx
  <TextControl(name=)>
  <SelectControl(code=[*LER, Lerwick, Scotland, 29.9 / 358.8, ESK, Eskdalemuir, Scotland, 34.7 / 356.8, HAD, Hartland, England, 39.0 / 355.5, ABN, Abinger, England, 38.8 / 359.6, GRW, Greenwich, England, 38.5 / 0.0])>
  <TextControl(month=)>
  <TextControl(year=)>
  <SubmitControl(<None>=Submit Query) (readonly)>
  <IgnoreControl(<None>=<None>)>>