Python 3 Beautiful Soup Web Scraping

Question

I'm currently working with BeautifulSoup. I seem to be having some issues related to encoding.

Here is my code:

import requests
from bs4 import BeautifulSoup
req = requests.get('https://pythonprogramming.net/parsememcparseface/')
soup = BeautifulSoup(req.content.decode('utf-8','ignore'))
print(soup.find_all('p'))

Here is my error:

 UnicodeEncodeError: 'ascii' codec can't encode character '\u1d90' in position 602: ordinal not in range(128)

Any help would be appreciated.

Answer 1

Please add "html5lib" or "html.parser"

#!/usr/bin/python
# -*- coding: utf-8 -*-

...

# Python 3.6.0
soup = BeautifulSoup(req.content.decode('utf-8','ignore'), "html5lib")

# Python 2.7.12
soup = BeautifulSoup(req.content.decode('utf-8','ignore'), "html.parser")

Answer 2

I tried to reproduce the issue that you are facing here but was not able to.

Here is what I tried.

>>> import requests
>>> from bs4 import BeautifulSoup

>>> req = requests.get('https://pythonprogramming.net/parsememcparseface/')

>>> soup = BeautifulSoup(req.content.decode('utf-8','ignore'))


Warning (from warnings module):
  File "C:\Python34\lib\site-packages\bs4\__init__.py", line 166
    markup_type=markup_type))
UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "html.parser")

>>> soup = BeautifulSoup(req.content.decode('utf-8','ignore'), 'html.parser')
>>> print(soup.find_all('p'))
[<p class="introduction">Oh, hello! This is a <span style="font-size:115%">wonderful</span> page meant to let you practice web scraping. This page was originally created to help people work with the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" target="blank"><strong>Beautiful Soup 4</strong></a> library.</p>, <p>The following table gives some general information for the following <code>programming languages</code>:</p>, <p>I think it's clear that, on a scale of 1-10, python is:</p>, <p>Javascript (dynamic data) test:</p>, <p class="jstest" id="yesnojs">y u bad tho?</p>, <p>Whᶐt hαppéns now¿</p>, <p><a href="/sitemap.xml" target="blank"><strong>sitemap</strong></a></p>, <p>
<a class="btn btn-flat white modal-close" href="#">Cancel</a>  
                        <a class="waves-effect waves-blue blue btn btn-flat modal-action modal-close" href="#">Login</a>
</p>, <p>
<a class="btn btn-flat white modal-close" href="#">Cancel</a>  
                                <button class="btn" type="submit" value="Register">Sign Up</button>
</p>, <p class="grey-text text-lighten-4">Contact: Harrison@pythonprogramming.net.</p>, <p class="grey-text right" style="padding-right:10px">Programming is a superpower.</p>]

Answer 3

I can duplicate your error message and eliminate troublesome characters.

First this code simply requests the page and attempts to save it. The attempt fails with the message you have seen. I create a copy of the page by converting it to bytes ignoring ugly character codes and then converting it back to characters. Now the page can be saved successfully.

I make soup with it and find the paragraph tags.

>>> from bs4 import BeautifulSoup
>>> import requests
>>> req = requests.get('https://pythonprogramming.net/parsememcparseface/').text
>>> open('c:/scratch/temp.htm', 'w').write(req)
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u1d90' in position 6702: character maps to <undefined>
>>> modReq = str(req.encode('utf-8', 'ignore'))
>>> open('c:/scratch/temp.htm', 'w').write(modReq)
12556
>>> soup = BeautifulSoup(modReq, 'lxml')
>>> paras = soup.findAll('p')
>>> len(paras)
12