如何将 CSS 选定字段转换为普通 python 字符串

Question

My scrapy project is giving me a strange encoding for items when using CSS selectors.

Here is the relevent code:

Once the scrapy request is made and the webpage is downloaded, parse_page is called with the response...

    def parse_page(self, response):

        # Using Selenium WebDriver to select elements
        records = self.driver.find_elements_by_css_selector('#searchResultsTable > tbody > tr')

        for record in records:

            # Convert selenium object into scrapy.Selector object (necessary to use .add_css methods)  
            sel = Selector(text=record.get_attribute('outerHTML'))

            # Instantiate RecordLoader (a custom item loader)
            il = RecordLoader(item=Record(), selector=sel)

            # Select element and pass to example_field's input processor
            il.add_css('example_field', 'td:nth-child(2) > a::text')

il.add_css() passes the result of the CSS selector to example_field's input processor which for demonstration purposes is only print statements and shows the issue...

def example_field_input_processor(text_html):
    print(text_html)
    print(type(text_html))
    print(text_html.encode('utf-8'))

Output:

'\xa0\xa004/29/2020 10:50:24 AM,\xa0\xa0\xa0'

<class 'str'>

b'\xc2\xa0\xc2\xa004/29/2020 10:50:24 AM,\xc2\xa0\xc2\xa0\xc2\xa0'

Here are my questions:

1) Why is it that the CSS selector didn't just give me a normal Python string? Does it have to do with the CSS selector casting to text with ::text . Is it because the webpage is in a different encoding? I checked if there was a <meta> tag that specified the site's encoding but there wasn't one.

2) When I force an encoding of 'utf-8' why don't I get a normal python string instead of a bytes string that shows all the Unicode characters?

3) My goal is to have just a normal python string (No prepended b, no weird characters) that I can parse. How?

Answer 1

While scraping you sometimes have to clean your results from unicode characters

They are usually as a result of spaces tabs and sometimes span

As a common practice clean all texts you scrape:

def string_cleaner(rouge_text):
    return ("".join(rouge_text.strip()).encode('ascii', 'ignore').decode("utf-8"))

Explaination:

Use split() and join to translate the characters and clear it of unicodes.

This part of the code "".join(rouge_text.strip())

Then encode it to ascii and decode it to utf-8 to remove special characters

This part of the code .encode('ascii','ignore').decode("utf-8"))

How you would apply it in your code

print(string_cleaner(text_html))