Getting AttributeError: 'DataFrame' object has no attribute 'shape'

Question

python

I am reading CSV into Pyspark Dataframe named 'InputDataFrame' using :

InputDataFrame = spark.read.csv(path=file_path,inferSchema=True,ignoreLeadingWhiteSpace=True,header=True)

After reading I am using :

InputDataFrame.schema.names 

to find column names. But I am getting below logs on console :

Traceback (most recent call last):

  File "/snap/pycharm-community/143/helpers/pydev/_pydevd_bundle/pydevd_xml.py", line 284, in frame_vars_to_xml
    xml += var_to_xml(v, str(k), evaluate_full_value=eval_full_val) 

  File "/snap/pycharm-community/143/helpers/pydev/_pydevd_bundle/pydevd_xml.py", line 384, in var_to_xml
    xml_shape = ' shape="%s"' % make_valid_xml_value(str(v.shape))  

  File "/home/ajinkya/.local/lib/python3.6/site-packages/pyspark/sql/dataframe.py", line 1300, in __getattr__
    "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))  

AttributeError: 'DataFrame' object has no attribute 'shape'  
Unexpected error, recovered safely.

Can anyone explain why is this happening ? Also is there any alternative way to find inferred schema of Pyspark Dataframe

''' Using Pycharm IDE for development '''

Answer 1

I had the same problem happening on some code that was working perfectly fine after migrating to the latest Pycharm version.

I assume you are using the latest Pycharm version (2019.2). I don't have an explanation to why this is causing the issue but installing the older Pycharm 2019.1.4 fixed the problem for me.

Answer 2

Agreed, run this on pycharm 2019.2 with no problem. Put a break point somewhere, debug and error will happen

    spark = SparkSession.builder.getOrCreate()
    pdf = pd.DataFrame({'A': ['asdf', 'fdsa'], 'B': [1, 2]})
    sdf = spark.createDataFrame(pdf)
    print(pdf)
    sdf.show()

Answer 3

I concur with Ben. The new version PyCharm code is expecting a Pandas DataFrame which contains a 'shape' attribute and not an RDD DataFrame (which does not). You will need to roll back to an older version or wait for JetBrains to issue a fix.