AWS Glue连接错误

Question

We have data(Tables) in S3 bucket(CSV) and need to apply join transformation and Store the result in S3. The Joining of tables is successful But the resulting Column List in S3 is Jumbled, order is not maintained in Output File. Also output file contains extra quotes(") and dots(.)

On applying only Mappings the order doesn't change ,Produces proper output without Jumbling.

Script is in python or Scala.

Script :

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job


args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "testdb", table_name = "table1", transformation_ctx = "datasource0")

datasource1 = glueContext.create_dynamic_frame.from_catalog(database = "testdb", table_name = "reftable", transformation_ctx = "datasource1")

datasource2 =datasource1.join(["aaaaaaaaaid"],["aaaaaaaaaid"],datasource0,transformation_ctx="join") 


datasink2 = glueContext.write_dynamic_frame.from_options(frame = datasource2, connection_type = "s3", connection_options = {"path": "s3://testing/Output"}, format = "csv", transformation_ctx = "datasink2")
job.commit()

Need Help !!!

Answer 1

Can you try this:

datasource2  = Join.apply(datasource0, datasource1, 'aaaaaaaaaid', 'aaaaaaaaaid')

This should work. Let me know, and also kindly accept/upvote the answer if this helped resolve the problem.

Regards

Yuva

Answer 2

I know this is old, Just mentioning the approach that worked for me.

1. Convert to data frames
2. Join them.

as mentioned in https://stackoverflow.com/a/54362245/8622986

Hope this helps.