[英]Extracting table data from web using python
I need to extract a table from a website "https://geniusimpex.org/pakistan-import-data/" which has like thousands of rows, so I wanted to automate the process using bs4 and selenium but when I extract the table only the table header is extracted.我需要从网站“https://geniusimpex.org/pakistan-import-data/”中提取一个表格,它有数千行,所以我想使用 bs4 和 selenium 来自动化这个过程,但是当我只提取表格时提取表 header。 This is the code I used
这是我使用的代码
from bs4 import BeautifulSoup
from urllib.request import urlopen
url = "https://geniusimpex.org/pakistan-import-data/"
html = urlopen(url)
soup = BeautifulSoup(html, 'lxml')
type(soup)
soup.prettify()
print(soup.find_all('tr'))
It shows the following output它显示了以下 output
[1]: https://i.stack.imgur.com/GItzv.png [1]: https://i.stack.imgur.com/GItzv.png
As you can see only first row is extracted.如您所见,仅提取了第一行。 Can someone please tell me why I am not able to extract the table and how can I do so?
有人可以告诉我为什么我无法提取表格,我该怎么做? It will be very helpful.
这将非常有帮助。 Sorry if I am not clear or couldn't explain my problem.
抱歉,如果我不清楚或无法解释我的问题。 This is the first time I am asking a question on stack overflow.
这是我第一次询问堆栈溢出问题。
The data is loaded from external URL as Json.数据从外部 URL 加载为 Json。 You can use this script to load the information:
您可以使用此脚本加载信息:
import json
import requests
url = 'https://geniusimpex.org/wp-admin/admin-ajax.php?action=ge_forecast_list_data&order=asc&offset={offset}&limit=1000'
offset = 0
while True:
data = requests.get(url.format(offset=offset)).json()
# print data to screen:
for row in data.get('rows', []):
for k, v in row.items():
print('{:<30} {}'.format(k, v))
print('-' * 80)
if len(data.get('rows', [])) != 1000:
break
offset += 1000
Prints:印刷:
...
--------------------------------------------------------------------------------
count T
importer_name <span file_id="27893" post_count="T" post_id="2157293">BISMILLAH STEEL FURNACE \n NEAR GRID STATION DEEWAN</span>
goods_description IRON AND STEEL REMELTABLE SCRAP HARMONIZED CODE: 7204.4990 REFERENCE NUMBER:UM/PAK/5146A ITN: X20200629019843 NWT WEIGHT-19.650 MT SHIPPERS LOAD, STOWAGE AND COUNT
hs_code
shipment_port NEWARK APT/NEW
gross_weight 19.65
number_of_packages 1
unit_of_packages PACKAGES
size_of_container 1 X 20FT
imported_from_name SEALINK INTERNATIONAL INC C/O\n UNIVERSAL METALS, ,
bill_of_lading_number SII145321
bill_of_lading_date <span data="10-08-2020">10-08-2020</span>
--------------------------------------------------------------------------------
count T
importer_name <span file_id="27938" post_count="T" post_id="2159597">ASAD SHAHZAD S/O FAQIR ZADA</span>
goods_description 1 USED VEHICLE TOYOTA VITZ CHASSIS NO: KSP130 -2204837
hs_code NA
shipment_port NAGOYA, AICHI
gross_weight .97
number_of_packages 1
unit_of_packages UNIT
size_of_container 1 X 40FT
imported_from_name KASHMIR MOTORS , 3055-9-104 KUZUTSUKA NIIGATA KITA
bill_of_lading_number TA200716H06- 10
bill_of_lading_date <span data="10-08-2020">10-08-2020</span>
--------------------------------------------------------------------------------
...
EDIT: To save to CSV, you can use this script:编辑:要保存到 CSV,您可以使用此脚本:
import json
import requests
import pandas as pd
url = 'https://geniusimpex.org/wp-admin/admin-ajax.php?action=ge_forecast_list_data&order=asc&offset={offset}&limit=1000'
offset = 0
all_data = []
while True:
data = requests.get(url.format(offset=offset)).json()
# print data to screen:
for row in data.get('rows', []):
all_data.append(row)
for k, v in row.items():
print('{:<30} {}'.format(k, v))
print('-' * 80)
if len(data.get('rows', [])) != 1000:
break
offset += 1000
df = pd.DataFrame(all_data)
df.to_csv('data.csv')
Produces:产生:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.