Scraping prices from Opensea

Question

I'm trying to scrape data from the website opensea, simply prices and token id for now. Thus I used requests and beautifulsoup in order to iterate through the different items of a collection and then put all the files. Problem is I get an error message regarding line 10 I don't understand. If anyone has an idea how I could do this I would be very glad. Regards

import requests
from bs4 import BeautifulSoup
bs = BeautifulSoup
datas = {}
for i in range(0, 20):
    p = str(i)
    url = "https://api.opensea.io/api/v1/asset/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d/" + p + "/"
    response = requests.get(url)
    tokenid = i
    a = bs.find("div", class_="Overflowreact__OverflowContainer-sc-10mm0lu-0 gjwKJf Price--amount")
    datas = {tokenid: a}

print(datas)

Answer 1

I am not sure what you are exactly looking for. I found some errors in your code, regardless what you try to achieve. I would use this code. However, all the items in data are empty, meaning I can't find any devs with that class.

import requests
from bs4 import BeautifulSoup as bs
datas = {}

# i and tokenid seem to be the same thing - I simplified it a bit
for tokenid in range(20):
    p = str(tokenid)
    url = "https://api.opensea.io/api/v1/asset/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d/" + p + "/"
    response = requests.get(url)
    html_code = bs(response.content, 'lxml')
    a = html_code.find("div", class_="Overflowreact__OverflowContainer-sc-10mm0lu-0 gjwKJf Price--amount")
    datas[tokenid] = a  # This is the way to set new items in the datas dict

Answer 2

I don't think you can scrape OpenSea data from their site. I attempted with BeautifulSoup and got nowhere because they have CloudFare protection. However, if you wanted to collect data like prices, you're better off using their API.

You may need to request an API key, but they don't take too long to get and it's fairly simple to request the data. Here's the link to their API docs [https://docs.opensea.io/reference/getting-assets][1]

Once you have that, the python code to request info for the entire doodle nft collection, for example would be something like this:

import requests

def collect_nft_data(nft_address:str, collection_size:int) -> dict:
    """This will get the data on all the nfts in a collection, so long as you input the correct 
    nft_address and collection size, and return it in a dictionary""" 

    #create a dictionary to store your data for each asset in the collection
    asset_data = dict()

    #call the api to collect data on each asset in the collection the token_id usually corresponds to a number i.e. there are 10000 assets in a collection and you want the first asset, the tokenID 
    #is usually 1. So in our loop, i will be the tokenID of each asset
    for i in range(1, collection_size+1):
        url = "https://api.opensea.io/api/v1/assets?token_ids=" + i + "&asset_contract_address=" + nft_address +"&order_direction=desc&offset=0&limit=20"

        headers = {"Accept": "application/json"}

        response = requests.request("GET", url, headers=headers)

        asset_data['token_' + str(i)] = response.text

#the address for the doodles collection
doodles_address = '0x8a90CAb2b38dba80c64b7734e58Ee1dB38B8992e'
#the collection size i.e. the num of nfts in the collection
num_assets=9999

asset_data = collect_nft_data(doodles_address, num_assets)

That will return all of the data though so you'd have to do some tweaking if you just wanted the price of each asset.

My closing point, is that OpenSea's software Developer Kit (SDK) is much more useful and allows for more freedom of interaction with their API. However, it's written in Typescript so might be a bit of a learning curve for you if you're new to coding.