如何将 tar.gz 文件直接从 URL 读取到 Pandas 中？

Question

The dataset I wish to read lives on GitHub as a tar.gz file and is updated every few hours. While I can always download this file, uncompress it, and read from CSV, it would be much better if I can directly read from this URL into a Pandas data frame in a timely manner.

After some Googling, I was able to download the compressed file and then read it as a data frame.

import requests
import tarfile
import pandas as pd

# Download file from GitHub
url = "https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true"
target_path = "latestdata.tar.gz"

response = requests.get(url, stream=True)
if response.status_code == 200:
    with open(target_path, "wb") as f:
        f.write(response.raw.read())

# Read from downloaded file
with tarfile.open(target_path, "r:*") as tar:
    csv_path = tar.getnames()[0]
    df = pd.read_csv(tar.extractfile(csv_path), header=0, sep=",")

However, I wonder if there is a way to directly read the file content into a data frame without first saving it locally. This may be useful if I want to build a web app later and don't have a local machine. Any help would be appreciated! Thanks!

Answer 1

You can use the BytesIO ( In-Memory Stream ) to keep the data in memory instead of saving the file to local machine.

Also As per the tarfile.open documentation , If fileobj is specified, it is used as an alternative to a file object opened in binary mode for name.

>>> import tarfile
>>> from io import BytesIO
>>>
>>> import requests
>>> import pandas as pd


>>> url = "https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true"
>>> response = requests.get(url, stream=True)
>>> with tarfile.open(fileobj=BytesIO(response.raw.read()), mode="r:gz") as tar_file:
...     for member in tar_file.getmembers():
...         f = tar_file.extractfile(member)
...         df = pd.read_csv(f)
...         print(df)

Answer 2

If you use ParData , this can be done pretty cleanly:

from tempfile import TemporaryDirectory

import pardata

schema = {
    'download_url': 'https://github.com/beoutbreakprepared/nCoV2019/blob/master/latest_data/latestdata.tar.gz?raw=true',
    'subdatasets': {
        'all': {
            'path': 'latestdata.csv',
            'format': {
                'id': 'table/csv'
            }
        }
    }
}

with TemporaryDirectory() as d:
    dataset = pardata.dataset.Dataset(schema=schema, data_dir=d)
    dataset.download(verify_checksum=False)
    my_csv = dataset.load()  # my_csv is a pandas.DataFrame object that stores the CSV file

print(my_csv)

Disclaimer: I'm a primary co-maintainer of ParData.