Python，大型csv文件上的pandas.read_csv，具有来自Google云端硬盘文件的1000万行

Question

I extracted a .csv file from Google Bigquery of 2 columns and 10 Million rows.

I have downloaded the file locally as a .csv with the size of 170Mb, then I uploaded the file to Google Drive, and I want to use pandas.read_csv() function to read it into pandas DataFrame in my Jupyter Notebook.

Here is the code I used, with specific fileID that I wanna read.

# read into pandasDF from .csv stored on Google Drive.
follow_network_df = pd.read_csv("https://drive.google.com/uc?export=download&id=1WqHWdgMVLPKVbFzIIprBBhe3I9faq4HA")

Then here is what I got:

It seems the 170Mb csv file is read as an html link?

While when I tried the same code with another csv file of 40Mb, it worked perfectly

# another csv file of 40Mb.
user_behavior_df = pd.read_csv("https://drive.google.com/uc?export=download&id=1NT3HZmrrbgUVBz5o6z_JwW5A5vRXOgJo")

Can anyone give me some hint on the root cause of the difference? Any ideas on how to read a csv file of 10 Million rows and 170Mb from online storage? I know it's possible to just read the 10 Million rows into pandasDF by just using the BigQuery interface or from local machine, but I have to include this as part of my submission, so it's only possible for me to read from online source.

Answer 1

The problem is that your first file is too large for Google Drive to scan for viruses, so there's a user prompt that gets displayed instead of the actual file. You can see this if you access the first file's link.

I'd say click on the user prompt and use the following url with pd.read_csv .