抓取表格时没有数据

Question

I am trying to scrape historical data from a table in coinmarketcap. However, the code that I run gives back "no data." I thought it would be fairly easy, but not sure what I am missing.

url = "https://coinmarketcap.com/currencies/bitcoin/historical-data/"

data = requests.get(url)

bs=BeautifulSoup(data.text, "lxml")
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    cols=[x.text.strip() for x in cols]
    print(cols)

Output:

C:\Users\Ejer\anaconda3\envs\pythonProject\python.exe C:/Users/Ejer/PycharmProjects/pythonProject/CloudSQL_test.py
['No Data']

Process finished with exit code 0

Answer 1

Your problem basically is you're trying to get a table but this table is dynamically created by JS in this case you need to call an interpreter for this JS. But however you just can check the.network monitor on your browser and you can get the requests and probably contains a full JSON or XML raw data and you don't need to scrape. I did it and I got this request:

https://web-api.coinmarketcap.com/v1/cryptocurrency/ohlcv/historical?id=1&convert=USD&time_start=1604016000&time_end=1609286400

Check it out and I hope help you!

Answer 2

You don't need to scrape the data, you can get request it:

import time
import requests


def get_timestamp(datetime: str):
    return int(time.mktime(time.strptime(datetime, '%Y-%m-%d %H:%M:%S')))


def get_btc_quotes(start_date: str, end_date: str):
    start = get_timestamp(start_date)
    end = get_timestamp(end_date)
    url = f'https://web-api.coinmarketcap.com/v1/cryptocurrency/ohlcv/historical?id=1&convert=USD&time_start={start}&time_end={end}'
    return requests.get(url).json()


data = get_btc_quotes(start_date='2020-12-01 00:00:00',
                      end_date='2020-12-10 00:00:00')

import pandas as pd
# making A LOT of assumptions here, hopefully the keys don't change in the future
data_flat = [quote['quote']['USD'] for quote in data['data']['quotes']]
df = pd.DataFrame(data_flat)

print(df)

Output:

           open          high           low         close        volume    market_cap                 timestamp
0  18801.743593  19308.330663  18347.717838  19201.091157  3.738770e+10  3.563810e+11  2020-12-02T23:59:59.999Z
1  19205.925404  19566.191884  18925.784434  19445.398480  3.193032e+10  3.609339e+11  2020-12-03T23:59:59.999Z
2  19446.966422  19511.404714  18697.192914  18699.765613  3.387239e+10  3.471114e+11  2020-12-04T23:59:59.999Z
3  18698.385279  19160.449265  18590.193675  19154.231131  2.724246e+10  3.555639e+11  2020-12-05T23:59:59.999Z
4  19154.180593  19390.499895  18897.894072  19345.120959  2.529378e+10  3.591235e+11  2020-12-06T23:59:59.999Z
5  19343.128798  19411.827676  18931.142919  19191.631287  2.689636e+10  3.562932e+11  2020-12-07T23:59:59.999Z
6  19191.529463  19283.478339  18269.945444  18321.144916  3.169229e+10  3.401488e+11  2020-12-08T23:59:59.999Z
7  18320.884784  18626.292652  17935.547820  18553.915377  3.442037e+10  3.444865e+11  2020-12-09T23:59:59.999Z
8  18553.299728  18553.299728  17957.065213  18264.992107  2.554713e+10  3.391369e+11  2020-12-10T23:59:59.999Z