pyspark using Pytest is not showing spark UI

Question

I have written a pytest case using pyspark (Spark 3.0) for reading a file and getting a count of data frames, but I am unable to see spark UI and I am getting an OOM Error. What is the solution and how to debug without seeing spark UI

Thanks,

Xi

Answer 1

Wrap all tests inside an shell script , and call that shell script in a python code

then call using spark-submit python.py