熊猫read_pickle缓慢

Question

I have Python 3.4 with Pandas 0.17. I noticed that my program takes ~30 seconds to read a pickle file.

df= pd.read_csv(a, skiprows=[1])
df.to_pickle(b)
df2 = pd.read_pickle(b)  --- This line takes almost 30 seconds.

The original csv file is ~185 MB (2967000 lines) and the pickle file is 125 MB.

I have another pickle file (~95 MB) which is working fine (can be read in <1 sec). Any suggestions?

Answer 1

I found a way to resolve the issue. My pickle file was being created by a cronjob from root user. Python program was developed in virtual environment. Global environment does not have pandas. So when the root user ran the cronjob, it created pickle file successfully but something was wrong with this file. I modified the cronjob to use python binary from my virtualenv and that that fixed the issue. I can see the size difference in pickle file created by global python and virtualenv python.

I am still not sure how root user was able to run the python file while it does not have pandas available.