How to debug an aws glue pyspark job

Question

I have a aws glue pyspark job which is long running after a certain command. In the log it is not writing anything after that command even a simple “print hello “ statement. How can I debug aws glue pyspark job which is long running and not even writing logs. Job is not throwing any error it shows running status in the console

Answer 1

AWS Glue is based on Apache Spark which means until an action called there will not be any actual execution. So if you put print statements in between and see them in the logs that does't mean that your job is executed up to that point. As your job is long running check this article by AWS which explains about Debugging Demanding Stages and Straggler Tasks . Also this is a good blog to take a look at.