使用Python / Pandas从大型csv文件获取相关矩阵时出现问题

Question

First of all: I'm a beginner with python and data analytics BUT I'm confident I understand the concepts enough so you don't have to over-simplify your answers.

My challenge is that I have to analyze huge chunks of machine data (timeseries over two years; 24 structure-identical csv-files, each with 170 columns, ~ 2.5 million rows, ~ 2.6gb size).

This data has to be analyzed in regard to correlations. The initally desired output is an 170x170 correlation matrix. Further analysis (lag, an asymetrical correlation matrix Input x Output) shall be postponed to the next step and is not primarily to be considered for your answer.

I've been able to read one of the files into a dataframe (using the IPython-Console of Spyder; for the cost of a lot of my 16gb memory).

import pandas as pd

df = pd.read_csv(r"C:\MyFilePath\...\TestData.csv", sep=';', encoding='iso-8859-1')

In[]: len(df.columns)
Out[]: 170

In[]: len(df)
Out[]: 2678401

But from there on I'm stuck...

The pandas.DataFrame.corr method does not work properly and returns (if it works) only a 10 x 10 Matrix with a lot of NaN values (which are in my understanding just a display for a non existent pearson correlation (close to or equal to zero)).

I have found several descriptions how to load data into my dataframe, which exceeds my RAM. Yet I was not able to fully understand the concept of loading chunks, especially in combination with my time series.

I would really appreciate, if you could provide me with a proper hint or snippet, so that I can solve this problem.

Ideally the result is, that I can run over all the csv-files and get the desired correlation matrix for all parameters.

Note: I am not bound to pandas. If you suggest another library which serves this problem in a better way, I'm happy to hear your solution. But due to the security policy of my company I am obliged to not download any additional software (or to be more precise: it is complicated...) The only other option I have at hand is MATLAB R2018.a

Answer 1

Pandas df.corr gives a correlation matrix NxN, where N is the number of columns. I tried it with 200 columns and it works.

The most likely reason is that your data are not clean. If pandas finds a data point that is not acceptable for the correlation operation, it excluded that column. Try to create a dataframe with only numbers and just one string in one of the fields and you'll see what I mean.

If the data are not in a good state, it would explain why there are so many nans as well. I think you have to do some cleaning and pre-processing on the data.