pySpark check if dataframe exists

Question

Is there a way to check if a dataframe exists in pySpark?

I know in native python, to check if dataframe exists:

exists(df_name) && is.data.frame(get(df_name))

How can this be done in pySpark? Since command exists throws an error.

Answer 1

It is same as Petel code. You can import the dataframe type.

 from pyspark.sql import DataFrame

 df= sc.parallelize([
 (1,2,3), (4,5,7)]).toDF(["a", "b", "c"])

 if df is not None and isinstance(df,DataFrame):
      #<some operation>
      print("dataframe exists")

Answer 2

try this: df_name is not None and isinstance(df_name, DataFrame)

Answer 3

I think you want to know if df_name is defined and pointing to a DataFrame . None of the answers above handle the case where df_name is not set. This does:

from pyspark.sql import DataFrame

try:
  if df_name is not None and isinstance(df_name,DataFrame):
    print('df_name exists')

except NameError as error:
  print('df_name does not exist and not defined')