AWS Python Shell-如何使用Glue目录连接

Question

I have a JDBC connection defined in Glue and I am able to use it successfully in a Glue Spark job. How would I use that same connection in a Glue Python Shell job? I can't find any templates how to do this though I've seen references that it is possible.

An alternative would be how to define a JDBC connection in the Python Shell where I would need to include an external library? pyodbc I've read is not available due to dependencies.

Answer 1

When you attach a JDBC connection to a Glue Python Shell job, it can only be used by Glue to launch ENIs in the specified subnet with the security groups. The jdbc url, username, password holds no value for the Python Shell job.

Thus, you'll have to provide an external package, like pymssql ( http://www.pymssql.org/en/stable/ ), and initialize the connection from the script itself.

You may refer the documentation for Providing Your Own Python Library: https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#create-python-egg-library

Answer 2

Here is code. Make sure same connection is added in your python shell job cloudformation template. We are using pg8000 library.

def get_connection(self, conn_name):

        client = boto3.client('glue', region_name=self.region_id)
        response = client.get_connection(Name=conn_name)
        print response

        connection_properties = response['Connection']['ConnectionProperties']
        URL = connection_properties['JDBC_CONNECTION_URL']
        url_list = URL.split("/")

        host = "{}".format(url_list[-2][:-5])
        port = url_list[-2][-4:]
        database = "{}".format(url_list[-1])
        user = "{}".format(connection_properties['USERNAME'])
        pwd = "{}".format(connection_properties['PASSWORD'])

        # print "user:{}".format(user)
        # print "pwd:{}".format(pwd)
        # print "host:{}".format(host)
        # print "port:{}".format(port)
        # print "database:{}".format(database)

        rs_conn = dbapi.connect(database=database, host=host, port=5439, \
                                user=user, password=pwd, ssl=True)
        cur = rs_conn.cursor()
        cur.execute("set statement_timeout = 1200000")
        rs_conn.commit()
        cur.close()
        return rs_conn