如何使用BeautifulSoup从html脚本中提取元素

Question

I'm new in Python programming and I'm using BeautifulSoup to do some web-scraping from Chile's county electoral department. My problem is this: I need to extract specific strings out from a script. After some cleaning, I obtain something like this:

<script type="text/javascript">
    document.writeln("<p align='left' class='cleleccion2008'>");
    document.writeln("&nbsp;&nbsp;&nbsp;&nbsp;<a href='geografico.htm'>&laquo;&nbsp;&nbsp;VOLVER MEN&Uacute;<\/a><br>");
    document.writeln("<\/p>");
    document.writeln("<div class='mapTitle'>REGI&Oacute;N<\/div>");
    document.writeln("<p align='left' class='cleleccion2008'>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'101'+")' >Regi&oacute;n I<\/a><br>");
    document.writeln("<\/p>");
    document.writeln("<br>");
    document.writeln("<div class='mapTitle'>COMUNAS<\/div>");
    document.writeln("<p align='left' class='cleleccion2008'>"); 
    if ( parent.DIR_ANO >= "2004"){
        document.writeln("  &nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2307'+")' >Alto Hospicio<\/a> <br>");
    }
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2101'+")' >Arica<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2102'+")' >Camarones<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2303'+")' >Cami&ntilde;a<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2304'+")' >Colchane<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2202'+")' >General Lagos<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2302'+")' >Huara<\/a><br>");  
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2301'+")' >Iquique<\/a><br>");  
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2305'+")' >Pica<\/a><br>");
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2306'+")' >Pozo Almonte<\/a><br>");   
    document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2201'+")' >Putre<\/a><br>");                
    document.writeln("<\/p>"); 
    document.close();                                                               
}
</script>

From this script, I want to extract, from the last 12 lines, the county name and code to create something like:

Code, County 2101, Arica 2102, Camarones ... 2201, Putre

Any help would be really appreciated. Thanks for all your responses/reads.

Answer 1

There is no specific js parser in BeautifulSoup , but it can be handled easily by using regex .

import re

text = '''
<script type="text/javascript">
    document.writeln("<p align='left' class='cleleccion2008'>");
    document.writeln("&nbsp;&nbsp;&nbsp;&nbsp;<a 
href='geografico.htm'>&laquo;&nbsp;&nbsp;VOLVER MEN&Uacute;<\/a><br>");
document.writeln("<\/p>");
document.writeln("<div class='mapTitle'>REGI&Oacute;N<\/div>");
document.writeln("<p align='left' class='cleleccion2008'>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'101'+")' >Regi&oacute;n I<\/a><br>");
document.writeln("<\/p>");
document.writeln("<br>");
document.writeln("<div class='mapTitle'>COMUNAS<\/div>");
document.writeln("<p align='left' class='cleleccion2008'>"); 
if ( parent.DIR_ANO >= "2004"){
    document.writeln("  &nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2307'+")' >Alto Hospicio<\/a> <br>");
}
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2101'+")' >Arica<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2102'+")' >Camarones<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2303'+")' >Cami&ntilde;a<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2304'+")' >Colchane<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2202'+")' >General Lagos<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2302'+")' >Huara<\/a><br>");  
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2301'+")' >Iquique<\/a><br>");  
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2305'+")' >Pica<\/a><br>");
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2306'+")' >Pozo Almonte<\/a><br>");   
document.writeln("  &nbsp;&nbsp;&bull;&nbsp;<a href='javascript:Consulta("+'2201'+")' >Putre<\/a><br>");                
document.writeln("<\/p>"); 
document.close();                                                               
}
</script>
'''

result_num = []
result_county = []
result = []

for i in re.findall('"[+]\'(.*?)\'[+]"', text):
    result_num.append(i)
for j in re.findall('\'[ ]>(.*?)<', text):
    if j != '':
        result_county.append(j)

result_county = result_county[2:]
result_num = result_num[2:]

for count in range(len(result_county)):
    result.append(result_county[count] + result_num[count])

print(result)

Output :

['Arica2101', 'Camarones2102', 'Camiña2303', 'Colchane2304', 'General Lagos2202', 'Huara2302', 'Iquique2301', 'Pica2305', 'Pozo Almonte2306', 'Putre2201']

Answer 2

Jihan is partially right, in that there are no javascript parsers available in BeautifulSoup explicitly. You will still likely need bs4 to perform the initial parsing. Regular expressions can help get you through the string parsing, but I would use a compiled regular expression rather than performing a re.findall() . Using re.findall() is likely to result in a lot of false positives and cleanup for you. If you perform the regex line-by-line, you can be much more confident you are grabbing the right data and perform validation as you iterate. It also ultimately leads to cleaner code and more manageable output.

Instead, you can explicitly pull the <script> tag out of the page contents and use str.splitlines() method on the script tag you want. This will split the entire tag into a list of strings. You might want to split on the ; character that signifies a javascript line termination so that it will work even in cases where you are dealing with "optimized" (obfuscated) javascript code that is obnoxiously smashed together.

At that point, you can use a compiled (or simple re.search() ) regex on each line. That way you are certain you are getting a line-by-line match. Here is the code.

import argparse
import bs4
import re
import requests


def parse_county_codes(soup_object):
    for tag in soup_object:
        tag = str(tag)
        lines = tag.splitlines()
        code_regex = re.compile('"[+]\'(.*?)\'[+]"')
        county_regex = re.compile('\'[ ]>(.*?)<')

        for line in lines:
            county = county_regex.search(line)
            code = code_regex.search(line)
            if county and code:
                print(county.group(1), ':', code.group(1))

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', '--input-file', dest='in_file', help='Input html')
    parser.add_argument('-u', '--url', dest='url', help='Some url\'s content you want to parse')
    args = parser.parse_args()

    if args.in_file:
        with open(args.in_file) as f:
            html_string = f.read()
            soup = bs4.BeautifulSoup(html_string, 'html.parser')
    elif args.url:
        try:
            # Remember to handle any possible url handling exceptions
            response = requests.get(args.url)
        except Exception as e:
            print("The following exception occurred while requesting the url\n{0}".format(args.url))
            print(e)
            return

        soup = bs4.BeautifulSoup(response.content, 'html.parser')
    else:
        print("Input missing. Please provide -i or -u")
        return

    script_tags = soup.find_all('script')
    parse_county_codes(script_tags)

if __name__ == '__main__':
    main()

The output of this code is as follows:

Región I : 101
Alto Hospicio : 2307
Arica : 2101
Camarones : 2102
Camiña : 2303
Colchane : 2304
General Lagos : 2202
Huara : 2302
Iquique : 2301
Pica : 2305
Pozo Almonte : 2306
Putre : 2201

Note there are some characters and escape-sequences for special characters that look out of place in the strings, but the regular expressions in their current form provided by Jihan are valid. If you want to clean up the output, you'd know best how to do that, so I'll leave that up to you. Be aware your mileage may vary when using regular expressions, and depending on the other web page contents, you can run into other problems.