表具有相同列名（具有不同数据）的Apache Spark Join数据集

Question

I want to join multiple datasets that have some columns with same name while having different data. This is possible to rename dataset columns while conversion to dataframe. But is it possible to use rename or setting prefix to column names while using datasets.

Dataset<Row> uct = spark.read().jdbc(jdbcUrl, "uct", connectionProperties);
Dataset<Row> si = spark.read().jdbc(jdbcUrl, "si", connectionProperties).filter("status = 'ACTIVE'");
Dataset<Row> uc = uct.join(si, uct.col("service_id").equalTo(si.col("id")))

uc will have columns with same name 'code' then it will be difficult to get value of code from either uct.code or si.code

Answer 1

Dataframe is an alias for Dataset. So practically you are using a dataframe in your code. If you want to retain both the columns with the same name, then you will have to rename one of the columns before performing join using "withColumnRenamed" option.