Jupyter ipython kernel 在大文件加载时死机

Question

I have a huge binary file of size ~10gbs which I want to load into a pandas dataframe on my Jupyter notebook. I am using the following code for creating the dataframe:

df = pd.DataFrame(np.fromfile('binary_file.dat', dtype = mydtype)) #the file has over 20 columns of dtype '<f8'

Everytime I run this command, my kernel dies. On debugging, I found that the np.fromfile command goes through but pd.dataframe command is the one which causes the crash. I am running this on a 4 core, 16 GB Ubuntu AWS server. I tried setting

os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'

as per a Stackoverflow answer, but it didn't help. How can I read this file without crashing the kernel? Is it possible to do without increasing the server RAM?

Any and all assistance is appreciated. Thank you

Answer 1

Try

df = pd.read_csv('.....\binary_file.dat' , sep="however you dat-file is separated",engine ='python')