stackoom
  简体   繁体   中英

How it is encode in javascript?

 原文  2020-06-10 12:30:52  0   1    javascript/ python/ web-scraping/ beautifulsoup

Question

I'm trying to scrape a webpage http://spys.one/en/anonymous-proxy-list/ to get a proxy address and port but as matter of fact, I can scrape address but not the port as it is contained inside javascript and encoded somehow take a look. I'm using requests and beautiful soup for that.

'''python
from bs4 import BeautifulSoup
import requests
url='http://spys.one/en/anonymous-proxy-list/'
r=requests.get(url)
soup=BeautifulSoup(r.content,'lxml')'''

in head of webpage

<script type="text/javascript">o5y5=7474;l2h8=6751;b2v2=5357;p6g7=4218;m3k1=2225;h8x4=1220;q7u1=5086;t0o5=4972;j0a1=1545;w3l2=1788;d4j0g7=0^o5y5;j0r8o5=1^l2h8;t0h8c3=2^b2v2;n4k1b2=3^p6g7;x4b2m3=4^m3k1;m3d4z6=5^h8x4;c3e5e5=6^q7u1;s9x4a1=7^t0o5;v2s9i9=8^j0a1;h8z6n4=9^w3l2;</script>

in body of webpage

document.write("<font class=spy2>:<\/font>"+(n4k1b2^p6g7)+(h8z6n4^w3l2)+(x4b2m3^m3k1)+(d4j0g7^o5y5)+(m3d4z6^h8x4))

the output result is ":3905" on the original webpage.

1 answers

solution1
 1 ACCPTED  2020-06-10 13:12:35

This is hard to do without rendering the page first, I was able to get the data in selenium:

url = 'http://spys.one/en/anonymous-proxy-list/'
driver.get(url)

table = [[col.text
          for col in row.find_elements_by_tag_name('td')]
         for row in driver.find_elements_by_xpath('//tr[contains(@class, "spy1x")]')]

data = [{k: v
         for k, v in zip(table[0], row)}
        for row in table[1:]]        

import json
print(json.dumps(data, indent=4))

Output: 

[
    {
        "Proxy address:port": "45.76.83.163:17001",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Germany (Frankfurt am Main)",
        "Hostname/ORG": "playbizz.ru (AS-CHOOPA)",
        "Latency**": "11.033",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "58% (7) -"
    },
    {
        "Proxy address:port": "103.117.23.33:33111",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "India (New Delhi)",
        "Hostname/ORG": "103.117.23.33 (Touch Net India Pvt. Ltd.)",
        "Latency**": "7.626",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "new -"
    },
    {
        "Proxy address:port": "124.219.176.139:39589",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Japan (Tokyo)",
        "Hostname/ORG": "p7cdbb08b.tokyff01.ap.so-net.ne.jp (So-net Entertainment Corporation)",
        "Latency**": "12.201",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "43% (255) -"
    },
    {
        "Proxy address:port": "2.186.13.72:30339",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Iran",
        "Hostname/ORG": "2.186.13.72 (Iran Telecommunication Company PJS)",
        "Latency**": "7.754",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "67% (2) +"
    },
    {
        "Proxy address:port": "40.119.154.217:80",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "Netherlands (Amsterdam) !!!",
        "Hostname/ORG": "40.119.154.217 (MICROSOFT-CORP-MSN-AS-BLOCK)",
        "Latency**": "0.224",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "new +"
    },
    {
        "Proxy address:port": "139.101.93.137:80",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Japan (Kizugawa)",
        "Hostname/ORG": "user-137093101139.kinet-tv.ne.jp (Kintetsu Cable Network Co., Ltd.)",
        "Latency**": "10.217",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "8% (7) -"
    },
    {
        "Proxy address:port": "101.51.141.46:37858",
        "Proxy type": "HTTPS",
        "Anonymity*": "HIA",
        "Country (city)": "Thailand (Bangkok)",
        "Hostname/ORG": "node-rvy.pool-101-51.dynamic.totinternet.net (TOT Public Company Limited)",
        "Latency**": "2.807",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "45% (72) -"
    },
    {
        "Proxy address:port": "45.235.163.35:33265",
        "Proxy type": "HTTPS",
        "Anonymity*": "HIA",
        "Country (city)": "Brazil !!!",
        "Hostname/ORG": "45-235-163-35.nortenettelecom.com.br (norte.net telecomunicacoes me)",
        "Latency**": "7.072",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "29% (20) -"
    },
    {
        "Proxy address:port": "51.68.228.131:80",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "France",
        "Hostname/ORG": "vps-7d8ac4c6.vps.ovh.net (OVH SAS)",
        "Latency**": "0.112",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "100% (44) +"
    },
    {
        "Proxy address:port": "15.165.112.193:3128",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "South Korea (Incheon) !!!",
        "Hostname/ORG": "ec2-15-165-112-193.ap-northeast-2.compute.amazonaws.com (AMAZON-02)",
        "Latency**": "5.233",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "54% (69) -"
    },
    {
        "Proxy address:port": "72.250.28.64:36851",
        "Proxy type": "HTTPS",
        "Anonymity*": "HIA",
        "Country (city)": "United States",
        "Hostname/ORG": "72.250.28.64 (T-MOBILE-AS21928)",
        "Latency**": "11.472",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "47% (421) -"
    },
    {
        "Proxy address:port": "217.61.21.73:3128",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "United Kingdom (Slough)",
        "Hostname/ORG": "host73-21-61-217.static.arubacloud.com (ArubaCloud Limited)",
        "Latency**": "0.24",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "100% (8) +"
    },
    {
        "Proxy address:port": "185.44.229.227:34930",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Armenia",
        "Hostname/ORG": "host-227.229.44.185.ucom.am (Ucom LLC)",
        "Latency**": "7.745",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "56% (254) -"
    },
    {
        "Proxy address:port": "176.63.205.248:54621",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Hungary (Sopron)",
        "Hostname/ORG": "catv-176-63-205-248.catv.broadband.hu (Liberty Global B.V.)",
        "Latency**": "8.906",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "29% (22) -"
    },
    {
        "Proxy address:port": "41.217.219.49:38114",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Malawi (Lilongwe)",
        "Hostname/ORG": "41.217.219.49 (SKYBAND)",
        "Latency**": "3.748",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "49% (21) -"
    },
    {
        "Proxy address:port": "202.154.180.53:48049",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Indonesia (Jakarta)",
        "Hostname/ORG": "202.154.180.53 (PT INDONESIA COMNETS PLUS)",
        "Latency**": "13.436",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "34% (10) -"
    },
    {
        "Proxy address:port": "170.81.141.249:37081",
        "Proxy type": "HTTPS",
        "Anonymity*": "HIA",
        "Country (city)": "Mexico (Aguascalientes)",
        "Hostname/ORG": "170.81.141.249 (NUEVA RED INTERNET DE MEXICO S DE RL DE CV)",
        "Latency**": "10.191",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "50% (3) -"
    },
    {
        "Proxy address:port": "197.89.175.235:8080",
        "Proxy type": "HTTP (Squid)",
        "Anonymity*": "ANM",
        "Country (city)": "South Africa (Wellington) !!!",
        "Hostname/ORG": "197-89-175-235.dsl.mweb.co.za (OPTINET)",
        "Latency**": "3.726",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "57% (17) -"
    },
    {
        "Proxy address:port": "159.203.82.173:3128",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "United States (Clifton)",
        "Hostname/ORG": "rdcproxy.sebpo.net (DIGITALOCEAN-ASN)",
        "Latency**": "15.455",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "37% (64) +"
    },
    {
        "Proxy address:port": "13.125.52.156:3128",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "South Korea (Incheon) !!!",
        "Hostname/ORG": "ec2-13-125-52-156.ap-northeast-2.compute.amazonaws.com (AMAZON-02)",
        "Latency**": "7.663",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "100% (2) +"
    },
    {
        "Proxy address:port": "37.57.15.43:33761",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Ukraine (Kharkiv)",
        "Hostname/ORG": "43.15.57.37.triolan.net (Content Delivery Network Ltd)",
        "Latency**": "11.894",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "52% (253) -"
    },
    {
        "Proxy address:port": "34.105.41.225:3128",
        "Proxy type": "HTTP (Squid)",
        "Anonymity*": "ANM",
        "Country (city)": "United States",
        "Hostname/ORG": "225.41.105.34.bc.googleusercontent.com (GOOGLE)",
        "Latency**": "9.458",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "67% (6) -"
    },
    {
        "Proxy address:port": "65.152.119.226:39408",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "United States (Denver)",
        "Hostname/ORG": "65.152.119.226 (CENTURYLINK-US-LEGACY-QWEST)",
        "Latency**": "3.001",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "22% (22) -"
    },
    {
        "Proxy address:port": "12.69.91.226:80",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "United States (Reno)",
        "Hostname/ORG": "12.69.91.226 (ATT-INTERNET4)",
        "Latency**": "10.76",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "35% (24) -"
    },
    {
        "Proxy address:port": "199.115.147.193:42760",
        "Proxy type": "HTTPS",
        "Anonymity*": "HIA",
        "Country (city)": "United States (Portsmouth) !!!",
        "Hostname/ORG": "199.115.147.193 (TELCOM-SYSTEMS)",
        "Latency**": "3.558",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "26% (8) +"
    },
    {
        "Proxy address:port": "125.26.108.170:61637",
        "Proxy type": "HTTP",
        "Anonymity*": "HIA",
        "Country (city)": "Thailand (Taling Chan)",
        "Hostname/ORG": "node-lgq.pool-125-26.dynamic.totinternet.net (TOT Public Company Limited)",
        "Latency**": "4.715",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "38% (151) +"
    },
    {
        "Proxy address:port": "34.125.152.121:3128",
        "Proxy type": "HTTPS (Squid)",
        "Anonymity*": "ANM",
        "Country (city)": "United States",
        "Hostname/ORG": "121.152.125.34.bc.googleusercontent.com (GOOGLE)",
        "Latency**": "12.601",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "67% (6) -"
    },
    {
        "Proxy address:port": "34.125.5.58:3128",
        "Proxy type": "HTTP (Squid)",
        "Anonymity*": "ANM",
        "Country (city)": "United States",
        "Hostname/ORG": "58.5.125.34.bc.googleusercontent.com (GOOGLE)",
        "Latency**": "12.049",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "60% (6) +"
    },
    {
        "Proxy address:port": "103.83.36.55:5836",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "United States (Asheville)",
        "Hostname/ORG": "dahit.substands.com (Medha Hosting)",
        "Latency**": "7.581",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "27% (4) -"
    },
    {
        "Proxy address:port": "3.136.226.213:3128",
        "Proxy type": "HTTP",
        "Anonymity*": "ANM",
        "Country (city)": "United States (Columbus)",
        "Hostname/ORG": "ec2-3-136-226-213.us-east-2.compute.amazonaws.com (AMAZON-02)",
        "Latency**": "8.172",
        "Speed***": "",
        "Uptime": "",
        "Check date (GMT+03)": "50% (9) +"
    }
]

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question How to encode ' character in Javascript? How to encode javascript function how to encode password in javascript? How to encode javascript in blogspot? How to encode string in javaScript How to encode string with javascript? How to encode the url in Javascript How to Encode JavaScript files? How to do not encode unicode, javascript How to encode the javascript in an efficient manner?
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM