从 Python 中的 HTML 页面获取表的前几行

Question

I am making a GET request to this website through python.

https://www.nhc.noaa.gov/gis/forecast/archive/?C=M;O=D

However, it downloads a HTML page with a huge table through the following python code

import requests
url = 'https://www.nhc.noaa.gov/gis/forecast/archive/?C=M;O=D'
r = requests.get(input_url_path)
url_list = r.text

This takes a lot of time and space.

Is there a way to download the first N rows of the table on this page?

Answer 1

Use streaming, and set your chunksize to get however much data you want back. You can iterate over the chunks until you get as many links as you want. It will probably go over by a few depending on the chunk size but it'll get you pretty close.

import requests
import re

n_rows = 100
url = 'https://www.nhc.noaa.gov/gis/forecast/archive/?C=M;O=D'
r = requests.get(url, stream=True)

links = []
with requests.get(url, stream=True) as r:
    for chunk in r.iter_content(chunk_size=500000):
        links.extend(re.findall(r'href="([^?\/].*?)"',str(x)))
        if len(links) >= n_rows:
                     break