BeautifulSoup未提取完整表格

Question

I need to extract proxy-data from html minimal example:

from bs4 import BeautifulSoup
import requests
url="http://www.gatherproxy.com/proxylist/country/?c=Germany"
r = requests.get(url,timeout=15)
sp=BeautifulSoup(r.content,'html.parser')
tab=sp.find("table",id="tblproxy")
rows=tab.find_all('tr')
print(rows)

the print command should show me all the content of the table... however,it seems like bs only parsed the first 2 rows, completely ignoring the rest...

any ideas?

Answer 1

It is b'coz the remaining rows are populated by javascript. You can use Requests-html or selenium to get this data. Alternatively you can also scrap them from the <script> tags.

from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('http://www.gatherproxy.com/proxylist/country/?c=Germany')
r.html.render()
sp=BeautifulSoup(r.html.html,'html.parser')
tab=sp.find("table",id="tblproxy")
rows=tab.find_all('tr')
print(rows[2:]) #only from third row

Output

[<tr class="proxy 212-66-152-231-1F90" country="Germany" port="8080" prx="212.66.152.231:8080" time="2019-01-28T01:07:14Z" tmres="578" type="Transparent"><td>2m 41s ago</td><td>212.66.152.231</td><td><a href="/proxylist/port/8080">8080</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">1479</span>/<span class="red">644</span></td><td class="cen normal">578ms</td></tr>, <tr class="proxy 89-163-152-248-1F90" country="Germany" port="8080" prx="89.163.152.248:8080" time="2019-01-28T01:07:10Z" tmres="43" type="Elite"><td>2m 45s ago</td><td>89.163.152.248</td><td><a href="/proxylist/port/8080">8080</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">1402</span>/<span class="red">254</span></td><td class="cen fast">43ms</td></tr>, <tr class="proxy 87-138-207-100-C38" country="Germany" port="3128" prx="87.138.207.100:3128" time="2019-01-28T01:06:37Z" tmres="200" type="Transparent"><td>3m 18s ago</td><td>87.138.207.100</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">4969</span>/<span class="red">1194</span></td><td class="cen fast">200ms</td></tr>, <tr class="proxy 87-138-222-119-C38" country="Germany" port="3128" prx="87.138.222.119:3128" time="2019-01-28T01:06:36Z" tmres="110" type="Transparent"><td>3m 19s ago</td><td>87.138.222.119</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">60</span>/<span class="red">10</span></td><td class="cen fast">110ms</td></tr>, <tr class="proxy 82-165-98-173-C38" country="Germany" port="3128" prx="82.165.98.173:3128" time="2019-01-28T01:06:20Z" tmres="51" type="Transparent"><td>3m 35s ago</td><td>82.165.98.173</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">85</span>/<span class="red">23</span></td><td class="cen fast">51ms</td></tr>, <tr class="proxy 178-63-63-229-C38" country="Germany" port="3128" prx="178.63.63.229:3128" time="2019-01-28T01:04:28Z" tmres="102" type="Transparent"><td>5m 27s ago</td><td>178.63.63.229</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">311</span>/<span class="red">126</span></td><td class="cen fast">102ms</td></tr>, <tr class="proxy 5-9-113-163-1775" country="Germany" port="6005" prx="5.9.113.163:6005" time="2019-01-28T01:04:08Z" tmres="269" type="Elite"><td>5m 47s ago</td><td>5.9.113.163</td><td><a href="/proxylist/port/6005">6005</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">51</span>/<span class="red">26</span></td><td class="cen fast">269ms</td></tr>, <tr class="proxy 5-9-113-163-1770" country="Germany" port="6000" prx="5.9.113.163:6000" time="2019-01-28T01:04:01Z" tmres="136" type="Elite"><td>5m 54s ago</td><td>5.9.113.163</td><td><a href="/proxylist/port/6000">6000</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">51</span>/<span class="red">7</span></td><td class="cen fast">136ms</td></tr>, <tr class="proxy 78-47-157-159-50" country="Germany" port="80" prx="78.47.157.159:80" time="2019-01-28T01:04:00Z" tmres="42" type="Elite"><td>5m 55s ago</td><td>78.47.157.159</td><td><a href="/proxylist/port/80">80</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">4631</span>/<span class="red">710</span></td><td class="cen fast">42ms</td></tr>, <tr class="proxy 88-99-242-188-C38" country="Germany" port="3128" prx="88.99.242.188:3128" time="2019-01-28T01:03:40Z" tmres="41" type="Transparent"><td>6m 15s ago</td><td>88.99.242.188</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">641</span>/<span class="red">237</span></td><td class="cen fast">41ms</td></tr>, <tr class="proxy 5-189-153-156-50" country="Germany" port="80" prx="5.189.153.156:80" time="2019-01-28T01:02:22Z" tmres="430" type="Elite"><td>7m 33s ago</td><td>5.189.153.156</td><td><a href="/proxylist/port/80">80</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">510</span>/<span class="red">370</span></td><td class="cen normal">430ms</td></tr>, <tr class="proxy 50-7-117-86-C38" country="Germany" port="3128" prx="50.7.117.86:3128" time="2019-01-28T01:02:21Z" tmres="52" type="Transparent"><td>7m 34s ago</td><td>50.7.117.86</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">3</span>/<span class="red">0</span></td><td class="cen fast">52ms</td></tr>, <tr class="proxy 88-99-35-242-1FB6" country="Germany" port="8118" prx="88.99.35.242:8118" time="2019-01-28T01:02:15Z" tmres="780" type="Elite"><td>7m 40s ago</td><td>88.99.35.242</td><td><a href="/proxylist/port/8118">8118</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">37</span>/<span class="red">21</span></td><td class="cen slow">780ms</td></tr>, <tr class="proxy 46-228-199-72-C38" country="Germany" port="3128" prx="46.228.199.72:3128" time="2019-01-28T01:01:43Z" tmres="31" type="Transparent"><td>8m 12s ago</td><td>46.228.199.72</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">11</span>/<span class="red">6</span></td><td class="cen fast">31ms</td></tr>, <tr class="proxy 46-101-208-13-50" country="Germany" port="80" prx="46.101.208.13:80" time="2019-01-28T01:01:31Z" tmres="37" type="Elite"><td>8m 24s ago</td><td>46.101.208.13</td><td><a href="/proxylist/port/80">80</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">731</span>/<span class="red">407</span></td><td class="cen fast">37ms</td></tr>, <tr class="proxy 46-101-138-229-1F90" country="Germany" port="8080" prx="46.101.138.229:8080" time="2019-01-28T01:01:30Z" tmres="34" type="Transparent"><td>8m 25s ago</td><td>46.101.138.229</td><td><a href="/proxylist/port/8080">8080</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">2</span>/<span class="red">0</span></td><td class="cen fast">34ms</td></tr>, <tr class="proxy 46-101-129-76-C38" country="Germany" port="3128" prx="46.101.129.76:3128" time="2019-01-28T01:01:26Z" tmres="31" type="Transparent"><td>8m 29s ago</td><td>46.101.129.76</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">2</span>/<span class="red">0</span></td><td class="cen fast">31ms</td></tr>, <tr class="proxy 178-63-238-172-C38" country="Germany" port="3128" prx="178.63.238.172:3128" time="2019-01-28T01:01:24Z" tmres="137" type="Elite"><td>8m 31s ago</td><td>178.63.238.172</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">77</span>/<span class="red">24</span></td><td class="cen fast">137ms</td></tr>, <tr class="proxy 95-216-24-3-C38" country="Germany" port="3128" prx="95.216.24.3:3128" time="2019-01-28T01:00:46Z" tmres="198" type="Elite"><td>9m 9s ago</td><td>95.216.24.3</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">1152</span>/<span class="red">799</span></td><td class="cen fast">198ms</td></tr>, <tr class="proxy 173-212-192-77-C38" country="Germany" port="3128" prx="173.212.192.77:3128" time="2019-01-28T00:58:31Z" tmres="848" type="Transparent"><td>11m 24s ago</td><td>173.212.192.77</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">2</span>/<span class="red">0</span></td><td class="cen slow">848ms</td></tr>, <tr class="proxy 78-47-157-159-22B8" country="Germany" port="8888" prx="78.47.157.159:8888" time="2019-01-28T00:58:28Z" tmres="46" type="Elite"><td>11m 27s ago</td><td>78.47.157.159</td><td><a href="/proxylist/port/8888">8888</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">3777</span>/<span class="red">451</span></td><td class="cen fast">46ms</td></tr>, <tr class="proxy 148-251-238-35-50" country="Germany" port="80" prx="148.251.238.35:80" time="2019-01-28T00:55:34Z" tmres="47" type="Elite"><td>14m 21s ago</td><td>148.251.238.35</td><td><a href="/proxylist/port/80">80</a></td><td><a class="proxy_elite" href="/proxylist/anonymity/?t=Elite">Elite</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">1352</span>/<span class="red">106</span></td><td class="cen fast">47ms</td></tr>, <tr class="proxy 213-136-85-70-C38" country="Germany" port="3128" prx="213.136.85.70:3128" time="2019-01-28T00:55:19Z" tmres="36" type="Transparent"><td>14m 36s ago</td><td>213.136.85.70</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">56</span>/<span class="red">14</span></td><td class="cen fast">36ms</td></tr>, <tr class="proxy 139-59-129-174-1F90" country="Germany" port="8080" prx="139.59.129.174:8080" time="2019-01-28T00:54:56Z" tmres="44" type="Transparent"><td>14m 59s ago</td><td>139.59.129.174</td><td><a href="/proxylist/port/8080">8080</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">1872</span>/<span class="red">422</span></td><td class="cen fast">44ms</td></tr>, <tr class="proxy 178-63-63-233-C38" country="Germany" port="3128" prx="178.63.63.233:3128" time="2019-01-28T00:54:56Z" tmres="225" type="Transparent"><td>14m 59s ago</td><td>178.63.63.233</td><td><a href="/proxylist/port/3128">3128</a></td><td><a class="proxy_transparent" href="/proxylist/anonymity/?t=Transparent">Transparent</a></td><td><a href="/proxylist/country/?c=Germany">Germany</a></td><td title=""></td><td class="cen"><span class="blue">217</span>/<span class="red">109</span></td><td class="cen fast">225ms</td></tr>]