PySpark Juypter 笔记本显示器

Question

I am quite new to jupyter notebook, and I am struggling to reveal my spark table (image 1) in a neater way similar to a normal panda dataframe (image 2). I am trying not to use pandas.

The display function doesnt work on spark dataframe:( Can someone help me? It will make it easier for me to analysis the spark table.

Answer 1

When you build your spark session, add the config.

.config("spark.sql.repl.eagerEval.enabled", True)

For example,

import findspark
findspark.init()

from pyspark.sql import SparkSession

spark = SparkSession.builder \
          .master("local") \
          .config("spark.sql.repl.eagerEval.enabled", True) \
          .getOrCreate()

and just call the dataframe without .show as follows

df

that will print the result in this way.