[英]How to solve 'org.apache.spark.sql.AnalysisException' exception
試圖在 Scala spark 中加入兩個數據集，即輸入和度量
它們看起來像這樣->
輸入：
+---------+----------+-----------------------------+------------------------+------+-------------------------+--------------------+----------+-----------------------------+------------------------+-----------------+----------+---------+----------+--------------------+---------+--------------------+-------------------+-------------------+--------------------+-------+-------------+------------+-----+------------+-------+-----------+-------------------+--------------------+----------------+--------------------+----------+-------------------+-------+---------+--------------------+---------------------+-------------+------------------+--------------------+--------------------+--------------------+--------------------+-------------+--------------+------------------+---------+----------+-------------+------------------+---------------+----------+----------+-----------------+----------------------------+------------+----------------+---------------+----------------------+---------------+-----------+-----------+---------------+--------------+--------+-----------+----------+
| seller|global_agg|global_agg_visible_impression|global_agg_keyword_click|sl_agg|sl_agg_visible_impression|sl_agg_keyword_click|sl_stg_agg|sl_stg_agg_visible_impression|sl_stg_agg_keyword_click|seller_seller_tag|adblock_id|ad_tag_id|browser_id| canonical_hash|device_id| domain_name| domain_pattern| global_pattern| google_url_category|hour_id|is_valid_suid|mobile_model|os_id|rc_num_calls|referer|screen_size| seasonal_pattern| seller_tag|seller_top_level| slot_id|state_code| url_pattern2| ctr|slideshow|is_commercial_url_nb|is_commercial_url_lda|keyword_click|visible_impression| rpm_url_part0| rpm_url_part1| rpm_url_part2| rpm_url_part3|sub_bidder_id|da_device_name| da_mobile_model|da_is_app|da_os_name|cm_os_version|cm_browser_version|da_browser_name|stats_date|learner_id|mobile_model_misc|rc_num_call_akamai_corrected|referer_misc|screen_size_misc|adblock_id_misc|seller_adblock_id_misc|seller_tag_misc|seller_misc|global_vl2r|global_vl2r_kwc|global_vl2r_vi| sl_vl2r|sl_vl2r_kwc|sl_vl2r_vi|
+---------+----------+-----------------------------+------------------------+------+-------------------------+--------------------+----------+-----------------------------+------------------------+-----------------+----------+---------+----------+--------------------+---------+--------------------+-------------------+-------------------+--------------------+-------+-------------+------------+-----+------------+-------+-----------+-------------------+--------------------+----------------+--------------------+----------+-------------------+-------+---------+--------------------+---------------------+-------------+------------------+--------------------+--------------------+--------------------+--------------------+-------------+--------------+------------------+---------+----------+-------------+------------------+---------------+----------+----------+-----------------+----------------------------+------------+----------------+---------------+----------------------+---------------+-----------+-----------+---------------+--------------+--------+-----------+----------+
|8CUS3H6GJ| 3199| 3199| 81| 139| 139| 5| 139| 139| 5| 8CUS3H6GJ__misc|1293334731|146589733| 5|3f1163154992c27f5...| 2| countryliving.com| .%2anews.%2a| .%2anews.%2a|People & Society ...| 23| 0| iPad| 6| 13| 1| 834x1112| .%2anews.%2a| gpt_lb_11| 8PRVCXX19|2dbb87b0ca5118f9c...| ma| .%2anews.%2a|4.9e-04| 1| NA| NA| 0| 1|countryliving.com...|countryliving.com...|countryliving.com...|countryliving.com...| 176| TABLET| IPAD| false| IOS| 14_8| 98.0.4758.97| CHROME MOBILE|2022030813| -1| iPad| 100| 1| misc| misc| 8CUS3H6GJ__misc| misc| 8CUS3H6GJ| 0.02532| 81| 3199|0.031515| 5| 139|
|8CU8ND892| 3199| 3199| 81| 2| 2| 0| 2| 2| 0| 8CU8ND892__misc|1303809593|816510261| 6|0fadd20698c8eb489...| 4| menupix.com| .%2amen.%2a| .%2amen.%2a|Food & Drink > Be...| 23| 0| Other| 6| 7| 1| NA| .%2amen.%2a| 300-AFR1-Menu| 8PRVCXX19|5e1617c92cc745abe...| mi| .%2amen.%2a|4.0e-04| 0| sensitive| NA| 0| 1|menupix.com_menud...|menupix.com_menu.php| menupix.com_NA| menupix.com_NA| 186| DESKTOP| SAFARI - OS X| false| OS X| 10_15_6| 15.3| SAFARI|2022030813| -1| Other| 7| 1| NA| misc| 8CU8ND892__misc| misc| 8CU8ND892| 0.02532| 81| 3199|0.024824| 0| 2|
|8CUS3H6GJ| 3199| 3199| 81| 139| 139| 5| 139| 139| 5| 8CUS3H6GJ__misc| 133963366|647881048| 16|02117a01ef3c83505...| 3| iseecars.com| .%2asale.%2a| .%2asale.%2a|Autos & Vehicles ...| 23| 0| iPhone| 6| 9| 1| 428x926| .%2asale.%2a|jam_srp-m-breaker-18| 8PRVCXX19| 647881048| fl| .%2asale.%2a|5.1e-04| 0| sensitive| NA| 0| 2|iseecars.com_cars...| iseecars.com_NA| iseecars.com_NA| iseecars.com_NA| 186| MOBILE PHONE|GENERIC_SMARTPHONE| true| IOS| 15_3_1| NA| SAFARI|2022030813| -1| iPhone| 9| 1| 428x926| misc| 8CUS3H6GJ__misc| misc| 8CUS3H6GJ| 0.02532| 81| 3199|0.031515| 5| 139|
|8CU5217S8| 3199| 3199| 81| 224| 224| 11| 224| 224| 11| 8CU5217S8__misc|1470166420|617251957| 6|97ad08b531a77a31a...| 3| usmagazine.com| .%2amagazine.%2a| .%2amagazine.%2a| Shopping| 23| 0| iPhone| 6| 1| 1| 375x667| .%2amagazine.%2a| NA| 8PRVCXX19|1632372f3fe8495e0...| az| .%2amagazine.%2a|3.5e-04| 0| NA| NA| 0| 1|usmagazine.com_ho...|usmagazine.com_gl...| usmagazine.com_NA| usmagazine.com_NA| 128| MOBILE PHONE| IPHONE| false| IOS| 15_1| 15.1| SAFARI|2022030813| -1| iPhone| 1| 1| 375x667| misc| 8CU5217S8__misc| misc| 8CU5217S8| 0.02532| 81| 3199|0.041765| 11| 224|
|8CU4V40B1| 3199| 3199| 81| 456| 456| 7| 456| 456| 7| 8CU4V40B1__misc|1614514466|454554330| 6|dbeed4c1cd0dc94bd...| 3|makingthymeforhea...| .%2amushroom.%2a| .%2amushroom.%2a| Food & Drink| 23| 0| iPhone| 6| 11| 1| 375x812| .%2amushroom.%2a|AdThrive_Content_...| 8PRVCXX19|77eca1e2d050c2dc0...| ga| .%2amushroom.%2a|0.0e+00| 0| NA| NA| 0| 1|makingthymeforhea...|makingthymeforhea...|makingthymeforhea...|makingthymeforhea...| 133| MOBILE PHONE| IPHONE| false| IOS| 14_8_1| 14.1.2| SAFARI|2022030813| -1| iPhone| 100| 1| 375x812| misc| 8CU4V40B1__misc| misc| 8CU4V40B1| 0.02532| 81| 3199|0.017144| 7| 456|
|8CU4V40B1| 3199| 3199| 81| 456| 456| 7| 456| 456| 7| 8CU4V40B1__misc|1645741457|867854626| 6|9a19532f0607f1a67...| 3| fitfoodiefinds.com| .%2achicken.%2a| .%2achicken.%2a|Food & Drink > Co...| 23| 0| iPhone| 6| 19| 1| 414x896| .%2achicken.%2a|AdThrive_Content_...| 8PRVCXX19|6c2626c75d0d61cea...| fl| .%2achicken.%2a|1.1e-04| 0| NA| NA| 0| 1|fitfoodiefinds.co...|fitfoodiefinds.co...|fitfoodiefinds.co...|fitfoodiefinds.co...| 196| MOBILE PHONE| IPHONE| false| IOS| 14_7_1| 14.1.2| SAFARI|2022030813| -1| iPhone| 100| 1| 414x896| misc| 8CU4V40B1__misc| misc| 8CU4V40B1| 0.02532| 81| 3199|0.017144| 7| 456|
|8CU65T3AT| 3199| 3199| 81| 9| 9| 0| 9| 9| 0| 8CU65T3AT__misc|1708947332|642241257| 6|e850b035360db101e...| 3| livestly.com| .%2ahousehold.%2a| .%2ahousehold.%2a|Home & Garden > H...| 23| 0| iPhone| 6| 1| 1| 414x736| .%2ahousehold.%2a|div-gpt-ad-149261...| 8PRVCXX19|3d94a0c408b09fc82...| la| .%2ahousehold.%2a|6.1e-04| 1| NA| NA| 0| 2|livestly.com_hous...| livestly.com_NA| livestly.com_NA| livestly.com_NA| 186| MOBILE PHONE| IPHONE| false| IOS| 14_7_1| 14.1.2| SAFARI|2022030813| -1| iPhone| 1| 1| misc| misc| 8CU65T3AT__misc| misc| 8CU65T3AT| 0.02532| 81| 3199|0.023229| 0| 9|
|8CUS3H6GJ| 3199| 3199| 81| 139| 139| 5| 139| 139| 5| 8CUS3H6GJ__misc|1763942358|647881048| 22|f12d9cd1c53bb4d29...| 3| caranddriver.com|.%2abest.%2asuv.%2a|.%2abest.%2asuv.%2a|Autos & Vehicles ...| 23| 0| iPhone| 6| 1| 1| 375x812|.%2abest.%2asuv.%2a| gpt_gal_a| 8PRVCXX19| 647881048| ma|.%2abest.%2asuv.%2a|9.4e-05| 0| sensitive| NA| 0| 1|caranddriver.com_...|caranddriver.com_...|caranddriver.com_...| caranddriver.com_NA| 186| MOBILE PHONE| IPHONE| true| IOS| 15_2_1| NA| SAFARI|2022030813| -1| iPhone| 1| 1| 375x812| misc| 8CUS3H6GJ__misc| misc| 8CUS3H6GJ| 0.02532| 81| 3199|0.031515| 5| 139|
|8CUXP6AUQ| 3199| 3199| 81| 19| 19| 0| 19| 19| 0| 8CUXP6AUQ__misc|1876213389|492267288| 6|2b2b8ce4d414d87c7...| 3| wunderground.com| .%2aforecast.%2a| .%2aforecast.%2a| News > Weather| 23| 0| Other| 6| 2| 1| NA| .%2aforecast.%2a| WX_Top300Variable| 8PRVCXX19|7d02cb34f1c34d1bf...| co| .%2aforecast.%2a|4.0e-03| 0| NA| NA| 0| 1|wunderground.com_...| wunderground.com_us| wunderground.com_co|wunderground.com_...| 128| DESKTOP| SAFARI - OS X| false| OS X| 10_15_6| 15.3| SAFARI|2022030813| -1| Other| 2| 1| NA| misc| 8CUXP6AUQ__misc| misc| 8CUXP6AUQ| 0.02532| 81| 3199|0.021277| 0| 19|
|8CUK5QD75| 3199| 3199| 81| 224| 224| 12| 224| 224| 12| 8CUK5QD75__misc|2026381580|373321055| 16|fbf8a2526836d2df7...| 3|thebestblogrecipe...| .%2arecipe.%2a| .%2arecipe.%2a|Food & Drink > Co...| 23| 1| iPhone| 6| 1| 1| 414x896| .%2arecipe.%2a| content_mobile| 8PRVCXX19|4ff74ea6b06d13aea...| mn| .%2arecipe.%2a|2.1e-03| 0| NA| NA| 1| 1|thebestblogrecipe...|thebestblogrecipe...|thebestblogrecipe...|thebestblogrecipe...| 196| MOBILE PHONE| IPHONE 11 PRO MAX| true| IOS| 15_2_1| NA| SAFARI|2022030813| -1| iPhone| 1| 1| 414x896| misc| 8CUK5QD75__misc| misc| 8CUK5QD75| 0.02532| 81| 3199|0.044852| 12| 224|
+---------+----------+-----------------------------+------------------------+------+-------------------------+--------------------+----------+-----------------------------+------------------------+-----------------+----------+---------+----------+--------------------+---------+--------------------+-------------------+-------------------+--------------------+-------+-------------+------------+-----+------------+-------+-----------+-------------------+--------------------+----------------+--------------------+----------+-------------------+-------+---------+--------------------+---------------------+-------------+------------------+--------------------+--------------------+--------------------+--------------------+-------------+--------------+------------------+---------+----------+-------------+------------------+---------------+----------+----------+-----------------+----------------------------+------------+----------------+---------------+----------------------+---------------+-----------+-----------+---------------+--------------+--------+-----------+----------+
only showing top 10 rows
公制
+---------+-----------------+------------------+-------------+--------+
| seller|seller_seller_tag|visible_impression|keyword_click| vl2r|
+---------+-----------------+------------------+-------------+--------+
|8CU5217S8| 8CU5217S8__misc| 224| 11|0.046841|
|8CUK5QD75| 8CUK5QD75__misc| 224| 12| 0.05088|
|8CUQJV5RI| 8CUQJV5RI__misc| 10| 1|0.038281|
|8CU29N1R8| 8CU29N1R8__misc| 6| 0|0.022535|
|8CUS47X5W| 8CUS47X5W__misc| 5| 1| 0.04156|
|8CU81SHO3| 8CU81SHO3__misc| 2| 0|0.024337|
|8CUWMI118| 8CUWMI118__misc| 1| 0|0.024821|
|8CU1NA8RS| 8CU1NA8RS__misc| 1| 0|0.024821|
|8CU6I65Y2| 8CU6I65Y2__misc| 10| 0|0.020925|
|8CUWMQE3H| 8CUWMQE3H__misc| 66| 1|0.018842|
|8CUJKX6Y3| 8CUJKX6Y3__misc| 4| 0| 0.02341|
|8CUYT9A1U| 8CUYT9A1U__misc| 1| 0|0.024821|
|8CU27488H| 8CU27488H__misc| 1| 0|0.024821|
|8CU7O5VP2| 8CU7O5VP2__misc| 4| 0| 0.02341|
|8CUS3H6GJ| 8CUS3H6GJ__misc| 139| 5|0.034107|
|8CUJN5H60| 8CUJN5H60__misc| 72| 0|0.008559|
|8CUHN3BGE| 8CUHN3BGE__misc| 63| 4|0.049125|
|8CUQ5LJ63| 8CUQ5LJ63__misc| 13| 0|0.019829|
|8CUM545EY| 8CUM545EY__misc| 23| 0|0.016736|
|8CU94FM32| 8CU94FM32__misc| 32| 0|0.014532|
+---------+-----------------+------------------+-------------+--------+
我要執行的操作->
input.as("input").join(broadcast(metric.as("metric"), Seq("seller","seller_seller_tag"), "left_outer")
此連接操作因錯誤而中斷：
Resolved attribute(s) seller_seller_tag#17561 missing from domain_pattern#10807,canonical_hash#10804,seller#10818,visible_impression#10829L,browser_id#10803,da_os_name#10838,domain_name#10806,slideshow#10825,os_id#10813,seasonal_pattern#10817,rpm_url_part1#10831,adblock_id#10801,da_browser_name#10841,da_mobile_model#10836,rpm_url_part3#10833,da_is_app#10837,seller_tag_misc#3625,referer#10815,seller_tag#10819,keyword_click#10828L,learner_id#10843,rpm_url_part2#10832,adblock_id_misc#2850,cm_os_version#10839,seller_misc#4023,global_pattern#10808,rc_num_calls#10814,screen_size#10816,google_url_category#10809,is_commercial_url_lda#10827,is_valid_suid#10811,url_pattern2#10823,stats_date#10842,sub_bidder_id#10834,da_device_name#10835,ad_tag_id#10802,hour_id#10810,seller_top_level#10820,mobile_model#10812,slot_id#10821,state_code#10822,device_id#10805,seller_adblock_id_misc#2899,screen_size_misc#2351,cm_browser_version#10840,rc_num_call_akamai_corrected#1385,is_commercial_url_nb#10826,ctr#10824,seller_seller_tag#3676,mobile_model_misc#1340,rpm_url_part0#10830,referer_misc#1866 in operator !Project [adblock_id#10801, ad_tag_id#10802, browser_id#10803, canonical_hash#10804, device_id#10805, domain_name#10806, domain_pattern#10807, global_pattern#10808, google_url_category#10809, hour_id#10810, is_valid_suid#10811, mobile_model#10812, os_id#10813, rc_num_calls#10814, referer#10815, screen_size#10816, seasonal_pattern#10817, seller#10818, seller_tag#10819, seller_top_level#10820, slot_id#10821, state_code#10822, url_pattern2#10823, ctr#10824, ... 28 more fields]. Attribute(s) with the same name appear in the operation: seller_seller_tag. Please check if the right attribute(s) are used.
顯然，“seller”和“seller_seller_tag”列都存在於左右數據集中。 錯誤消息含糊不清，幾天以來我一直在為此苦思冥想。 嘗試了幾件事，包括只選擇幾列，只加入“賣家”列。 一切都是徒勞
我無法重現您的錯誤，您是否在加入后將您實施的所有轉換分享給 dataframe？
嘗試重命名有問題的列，然后加入新的字段名稱：
val metricRenamed = metric.withColumnRenamed("seller_seller_tag", "metric_seller_seller_tag")
input.join(broadcast(metricRenamed), input("seller") === metricRenamed("seller") && input("seller_seller_tag") === metricRenamed("metric_seller_seller_tag"), "left_outer")
線程“ main” org.apache.spark.sql.AnalysisException中的異常：
[英]Exception in thread “main” org.apache.spark.sql.AnalysisException:
診斷：用戶 class 拋出異常：org.apache.spark.sql.AnalysisException：路徑 {PATH} 已存在
[英]diagnostics: User class threw exception: org.apache.spark.sql.AnalysisException: path {PATH} already exists
如何在更改數據框中的列順序時修復org.apache.spark.sql.AnalysisException？
[英]How to fix org.apache.spark.sql.AnalysisException while changing the order of columns in a dataframe?
線程“main”org.apache.spark.sql.AnalysisException 中的異常：由於數據類型不匹配，無法解析“named_struct()”：
[英]Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'named_struct()' due to data type mismatch:
線程“主”org.apache.spark.sql.AnalysisException 中的異常：無法解析給定輸入列的“id”：[0、33、385、Will]；
[英]Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`id`' given input columns: [0, 33, 385, Will];
Spark ML-KMeans-org.apache.spark.sql.AnalysisException：無法解析給定輸入列的“功能”
[英]Spark ML - KMeans - org.apache.spark.sql.AnalysisException: cannot resolve '`features`' given input columns
Spark DeltaLake Upsert（合並）正在拋出“org.apache.spark.sql.AnalysisException”
[英]Spark DeltaLake Upsert (merge) is throwing “org.apache.spark.sql.AnalysisException”
Spark 流式傳輸 2.4.0 獲取 org.apache.spark.sql.AnalysisException：找不到數據源：kafka
[英]Spark streaming 2.4.0 getting org.apache.spark.sql.AnalysisException: Failed to find data source: kafka
加入 scala 火花時出錯。 org.apache.spark.sql.AnalysisException：缺少解析的屬性
[英]Error with join in scala spark. org.apache.spark.sql.AnalysisException: Resolved attribute(s) missing
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.