AWS Glue PySpark can't count the records

Question

I'm using AWS Glue to extract data from EC2 (Postgre) to be transformed and put it on S3 when I tried to extract 1 table. I got an error looks like this:

Is there anything I can do? I tried to drop null fields or fillna, but none of those works.

UPDATE: I even selected a string-type column but still got the same error:

Answer 1

Can you try, df.isnull().any() or df.isnull().sum() . This should help us see the columns with invalid NaN data. Also please try to fetch count of records with df.count(dropna = False) / df.na.drop() . Please refer here , where its explained more in detail on handling null column data.

Hope this helps.