How do I download multiple text files from Google Drive and append to Pandas data frame?

Question

I'm trying to download 4 text files from Google drive into a single Pandas data frame for analysis. Here is my code:

# Import Pandas and other stuff
import pandas as pd
import numpy as np
import datetime as dt
from matplotlib import pyplot as plt

# Setup Google Drive access - code to read csv file into Colaboratory:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# Import weather data from Google Drive
dataFiles = [['https://drive.google.com/open?id=1w3CRxNbIYDXhEgkqwn8BB78C9O2WWLKi','Environmental_Data_Deep_Moor_2012.txt'],
             ['https://drive.google.com/open?id=1_aHbOnVIOHWUMjIKY9cL3w-0qbwqtZRE','Environmental_Data_Deep_Moor_2013.txt'],
             ['https://drive.google.com/open?id=1cQOB_jdOEgOtjq1qllBsagGRSzKW_Nii','Environmental_Data_Deep_Moor_2014.txt'],
             ['https://drive.google.com/open?id=17f-0D0y_n4PpAu_M674amFYL9AnExLod','Environmental_Data_Deep_Moor_2015.txt']]

# Create empty array for file ID numbers
fileIDs =[]

# Split up the file URL to fetch the file ID and download into dataframes
for i in range(0,len(dataFiles)):
  fluff, id = dataFiles[i][0].split('=')
  fileIDs.append(id)

  # If this is the first file being loaded, create a new dataframe, otherwise append:
  downloaded[i] = drive.CreateFile({'id':id})
  downloaded[i].GetContentFile(dataFiles[i][1])
  df_append = pd.read_csv(dataFiles[i][1], sep="\t")
  df_weather.append(df_append)
  df_append.head()
  print("File ID: {} loaded. There are {} total lines loaded into the df_weather data frame.".format(fileIDs[i],len(df_weather)))

It seems only the first file is being loaded into the data frame. Any ideas why the subsequent files aren't being loaded?

Answer 1

Found the problem... I needed to assign the df_append data frame back to the df_weather data frame. Here is my code:

# Create empty array for file ID numbers and and empty data frame for the
# weather data with the df_weather data frame
fileIDs =[]
df_weather = pd.DataFrame()

# Split up the file URL to fetch the file ID and download into dataframes
for i in range(0,len(dataFiles)):
  fluff, id = dataFiles[i][0].split('=')
  fileIDs.append(id)

  # If this is the first file being loaded, create a new dataframe, otherwise append:
  downloaded = drive.CreateFile({'id':id})
  downloaded.GetContentFile(dataFiles[i][1])
  df_append = pd.read_csv(dataFiles[i][1], sep="\t")
  df_weather = df_weather.append(df_append)
  print("File ID: {} loaded. There are {} total lines loaded into the df_weather data frame.".format(fileIDs[i],len(df_weather)))