Python-使用request和bs4进行超级简单的抓取

Question

I am trying to get the data from the main table from this page: https://www.interactivebrokers.com/en/index.php?f=2222&exch=globex&showcategories=FUTGRP#productbuffer

I tried:

import requests
from bs4 import BeautifulSoup

address="https://www.interactivebrokers.com/en/index.php?f=2222&exch=globex&showcategories=FUTGRP#productbuffer"

r=requests.get(address)
soup=(r.text,"html_parser")

I know this is super basic but somehow i'm blocked here.

I tried soup.find_all('table') but couldn't identify correctly the table i'm looking for (it seems to have no ID or distinguishable attribute).

I tried soup.find_all('tr') then i can see the rows i am looking for but there is also some undesired rows in the result that i don't know how to separate.

Anyone can help me with my first step with bs4?

Answer 1

It seems the problem is, that the data you want actually resides outside the table tag, but in a tbody-tag. The site has 3 of these.

So a working code to grab the tds would look like this:

import requests
from bs4 import BeautifulSoup

url = 'https://www.interactivebrokers.com/en/index.php?f=2222&exch=globex&showcategories=FUTGRP#productbuffer'
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
table = soup.find_all('tbody')[2]
trs = table.find_all('tr')

Then you just have to iterate over the trs to get the content, you're after. tds are in a list which has four elements. You are after nr. 0, 2 and 3. Normally you could go with that. Since nr 1 has the same link text ('linkexternal') I went by that instead.

outfile = r'C:\output_file.txt'
with open(outfile, 'a', encoding='utf-8') as fd:
    for tr in trs:
        try:
            tds = tr.find_all('td')
            print_elements = ",".join([td.text for td in tds if 'linkexternal' not in str(td)])
            fd.write(print_elements+'\n')
        except:
            #some exception handling, perhaps logging
            pass