emr-container pyspark job running indefinitely

Question

Here's my Python script:

import calendar
import pydeequ
import boto3
import psycopg2
import os
import pyspark

from py4j import *
from pyspark.sql import SparkSession,Row
from pydeequ.profiles import *
from pydeequ.suggestions import *
from pydeequ.repository import *
from pydeequ.analyzers import *
from pyspark.sql import SparkSession
from botocore.config import Config
from datetime import datetime,timedelta,date
from pyspark.conf import SparkConf
from pydeequ.checks import *
from pydeequ.verification import *
from py4j.java_gateway import java_import

print(os.system("""pyspark --version"""))

spark = (SparkSession.builder \
        .appName('run_dq_for_xpertrak_pathtrak') \
        .enableHiveSupport() \
        .config(conf=SparkConf()) \
        .config("spark.jars.packages", pydeequ.deequ_maven_coord) \
        .config("spark.jars.excludes", pydeequ.f2j_maven_coord) \
        .getOrCreate())

java_import(spark._sc._jvm, "org.apache.spark.sql.*")

print('here---')
print(spark)

junk = spark.sql("""SELECT * FROM xpertrak.pathtrak LIMIT 10""")

print(junk)

Within AWS emr-containers (ie EMR on EKS ), this job successfully runs and UI shows that indeed the job completed. However, when I include or append the following lines of code to the bottom of script above, the job technically completes (based on simple logs prints), but the UI never changes from the running state...

print('checking')
check = Check(spark, level=CheckLevel.Warning, description="Data Validation Check")
checkResult = VerificationSuite(spark) \
    .onData(junk) \
    .addCheck(
        check.hasSize(lambda x: x >= 5000000)
    ).run()
print(checkResult)
print('check')

This is what that looks like the AWS console/UI:

What could be causing this anomaly?

Answer 1

Based on AWS-supplied docs from here , adding the following ended the job successfully:

spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()