Analyzing data from multiple .txt files in pandas

Question

I have 1000+ text files. Each has dates ( which I have made the index) and stock prices (which are column 0). I have created the code to find an individual file's price's moving average, and rolling difference between the price and the moving average. I would like to create code to do this for every file. I have to upload them in groups because it uses too much memory to upload them at once.

I imagine I would have to use a for loop to iterate through the files and find the metrics for each one. But how would I do that? How can I upload all the files into a group, and say, group them into one variable, then create a loop to find the moving average and difference from price for each one?

Edit: I am using numpy,pandas, and matplotlib. I'd also like to be able to find the stocks which the difference from the moving average is the greatest.

Any help would be greatly appreciated

Answer 1

If you are looking to just iterate over all of your input files in a given folder, you might want to try os.listdir() to get a list of filenames, which you can then process sequentially. If your files are spread over layers of folder, you could use os.walk() to traverse the directories. You can find info on these methods here: https://docs.python.org/3/library/os.html

Answer 2

How large are these 1000files? If they are a couple MB each, just guessing, merge all files into one single file and you can do whatever you want with it.

import pandas as pd
import csv
import glob
import os

#os.chdir("C:\\Users\\Excel\\Desktop\\test\\")
results = pd.DataFrame([])
filelist = glob.glob("C:\\your_path\\*.csv")
#dfList=[]
for filename in filelist:
    print(filename)  
    namedf = pd.read_csv(filename, skiprows=0, index_col=0)
    results = results.append(namedf)

results.to_csv('C:\\your_path\\CombinedFile.csv')