网站上的Beautifulsoup刮桌，要求提供熊猫

Question

I am trying to download the data on this website https://coinmunity.co/ ...in order to manipulate later it in Python or Pandas I have tried to do it directly to Pandas via Requests, but did not work, using this code:

res = requests.get("https://coinmunity.co/")
soup = BeautifulSoup(res.content, 'lxml')
table = soup.find_all('table')[0]
dfm = pd.read_html(str(table), header = 0)
dfm = dfm[0].dropna(axis=0, thresh=4)
dfm.head()

In most of the things I tried, I could only get to the info in the headers, which seems to be the only table seen in this page by the code.

Seeing that this did not work, I tried to do the same scraping with Requests and BeautifulSoup, but it did not work either. This is my code:

import requests
from bs4 import BeautifulSoup

res = requests.get("https://coinmunity.co/")
soup = BeautifulSoup(res.content, 'lxml')
#table = soup.find_all('table')[0]
#table = soup.find_all('div', {'class':'inner-container'})
#table = soup.find_all('tbody', {'class':'_ngcontent-c0'})
#table = soup.find_all('table')[0].findAll('tr')
#table = soup.find_all('table')[0].find('tbody')#.find_all('tbody _ngcontent-c3=""')
table = soup.find_all('p', {'class':'stats change positiveSubscribers'})

You can see in the lines commented, all the things I have tried, but nothing worked. Is there any way to easily download that table to use it on Pandas/Python, in the tidiest, easier and quickest possible way? Thank you

Answer 1

Since the content is loaded dynamically after the initial request is made, you won't be able to scrape this data with request. Here's what I would do instead:

from selenium import webdriver
import pandas as pd
import time
from bs4 import BeautifulSoup

driver = webdriver.Firefox()
driver.implicitly_wait(10)
driver.get("https://coinmunity.co/")

html = driver.page_source.encode('utf-8')

soup = BeautifulSoup(html, 'lxml')

results = []
for row in soup.find_all('tr')[2:]:
    data = row.find_all('td')
    name = data[1].find('a').text
    value = data[2].find('p').text
    # get the rest of the data you need about each coin here, then add it to the dictionary that you append to results
    results.append({'name':name, 'value':value})

df = pd.DataFrame(results)

df.head()

name    value
0   NULS    14,005
1   VEN 84,486
2   EDO 20,052
3   CLUB    1,996
4   HSR 8,433

You will need to make sure that geckodriver is installed and that it is in your PATH. I just scraped the name of each coin and the value but getting the rest of the information should be easy.