使用Python对Web内容进行Web抓取

Question

I'm trying to get the data of 'SALES HISTORY' from here .

Since the data is coming from JavaScript, I refereed to this link and tried to scrape the data. However when i run the below code, the new window doesn't show the web page properly.

I would be appreciated if you could advice how to get the data in this case.

# import libraries
import urllib.request
from bs4 import BeautifulSoup
from selenium import webdriver
import time
import pandas as pd
# specify the url
urlpage = 'https://nonfungible.com/market/history/axieinfinity' 
print(urlpage)
# run Chrome webdriver from executable path of your choice
driver = webdriver.Chrome(executable_path = r'C:\Users\trey\AppData\Local\Programs\Python\Python36\Scripts\chromedriver')

I expect the output of the data set which contains Contract/Transaction hash/Seller/Buyer/Name/Birth Date columns.

Answer 1

You don't need to scrape the site to get the sales history data, as you can get it from their JSON API end-point.

Here's the link to the end-point from the web page you posted:

https://api.nonfungible.com/api/v3/project/list

You can use Python JSON library to extract the data that you want. To find whether a site has a usable JSON API, use the network monitor from your browser developer console to find the XHR requests made to the site, and check if it contains the data you require. This would make more sense than scraping the HTML/JS.