使用左外部联接火花无法联接数据帧

Question

I am trying to join 2 dataframes based on df1.portfolio name to df2.portId the resulting dataframe I do not want the same key repeated.

here is my code so far

val df = spark.read.json("C:\\json\\portmast") 
val pgetsec = spark.read.json("C:\\json\\pgetsec")


val portfolio_master = df.select("PortfolioCode","Legal Entity Name","Asofdate")
val pgetsecs= pgetsec.select("TransId", "SecId","portId","GaapCurBkBal","ParBal","SetlDt","SetlPric","OrgBkBal","TradeDt","StatCurBkBal","NaicRtg","SecurityTypeCode","CamraSecType","FundType","CountryIso")
val pg = portfolio_master.join(pgetsec,Seq("PortfolioCode","portId"),"left_outer")

the error I am getting is
Exception in thread "main" org.apache.spark.sql.AnalysisException: using columns ['PortfolioCode,'portId] can not be resolved given input columns: Final json should look like this

|-- Portfolio Code: string (nullable = true)
|-- Legal Entity Name: string (nullable = true)
|-- Asofdate: string (nullable = true)

((SI, S&P 500 Index,9/30/2016),[0.0,Equity,Common Stock])
((SI, S&P 500 Index,9/30/2016),[0.0,Equity,Common Stock])
((SI, S&P 500 Index,9/30/2016),[0.0,Equity,Common Stock])
[SI1, S&P 500 Index,9/30/2016,CompactBuffer([0.0,Equity,Common     Stock],    [0.0,Equity,Common Stock], [0.0,Equity,Common Stock])]
root
|-- Portfolio Code: string (nullable = true)
|-- Legal Entity Name: string (nullable = true)
|-- Asofdate: string (nullable = true)
|-- Security: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- BondPrice: double (nullable = true)
|    |    |-- CoreSectorLevel1Code: string (nullable = true)
|    |    |-- CoreSectorLevel2Code: string (nullable = true)

+--------------+-------------------+---------+--------------------+
|Portfolio Code|  Legal Entity Name| Asofdate|            Security|
+--------------+-------------------+---------+--------------------+
|           SI | S&P 500 Index     |9/30/2016|[[0.0,Equity,Comm...|
+--------------+-------------------+---------+--------------------+

Any help is appreciated.

Answer 1

portId doesn't exist in portfolio_master and PortfolioCode doesn't exist in pgetsec . If you reread the full error message you'll see it explains this as it also shows the available columns.

What you want is portfolio_master("PortfolioCode") === pgetsec("portId") as your join condition.