Pandas Read_Parquet NaN error: ValueError: cannot convert float NaN to integer

Question

I need help in handling the NaN error I am getting in reading a parquet file in Python3. There is a column returning " NaN " while I try to read the file. Per requirement, I cannot change the content of this file - So I cannot go in and handle the offending column.

Here is the code I am using to read the file:

df = pd.read_parquet("parquet_file.parquet")

Here is the error:

ValueError: cannot convert float NaN to integer

Thanks.

Answer 1

Having come across a similar issue, I found the solution (in my case) was to pip install pyarrow. The documentation for read_parquet mentions the engine argument. auto is default, using the fastparquet library if pyarrow is not available.

pip installing pyarrow then rerunning the code seems to solve the issue

Answer 2

pandas.read_parquet(parquet_file.parquet, columns=[column1,column2])

You can specify the list of columns you want to process from the file exluding the column which is having issue