I'm looking for a way to combine two DataFrames by key. I started by creating Dataframes from rdds :
Given :
x = sc.parallelize([('_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', 'FR', '75001'),
('_guid_XblBPCaB8qx9SK3D4HuAZwO-1cuBPc1GgfgNUC2PYm4=', 'TN', '8160'),
]
)
y = sc.parallelize([('_guid_oX6Lu2xxHtA_T93sK6igyW5RaHH1tAsWcF0RpNx_kUQ=', 'JmJCFu3N'),
('_guid_hG88Yt5EUsqT8a06Cy380ga3XHPwaFylNyuvvqDslCw=', 'KNPQLQth'),
('_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', 'KlGZj08d'),
]
)
My code :
df_x = x.toDF(['id', 'countrycode', 'postalcode'])
df_y = y.toDF(['id_gigya', 'krux'])
df = df_x.join(df_y, df_x.id == df_y.id_gigya, 'fullouter')
which gives :
[Row(id=u'_guid_XblBPCaB8qx9SK3D4HuAZwO-1cuBPc1GgfgNUC2PYm4=', countrycode=u'TN', postalcode=u'8160', id_gigya=None, krux=None),
Row(id=None, countrycode=None, postalcode=None, id_gigya=u'_guid_oX6Lu2xxHtA_T93sK6igyW5RaHH1tAsWcF0RpNx_kUQ=', krux=u'JmJCFu3N'),
Row(id=None, countrycode=None, postalcode=None, id_gigya=u'_guid_hG88Yt5EUsqT8a06Cy380ga3XHPwaFylNyuvvqDslCw=', krux=u'KNPQLQth'),
Row(id=u'_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', countrycode=u'FR', postalcode=u'75001', id_gigya=u'_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', krux=u'KlGZj08d')]
It's perfect, but I want a keep a unique id, either 'id_gigya' or 'id', since it's the same id !
With :
df_x.join(df_y, df_x.id == df_y.id_gigya, 'fullouter').drop(df_y.id_gigya).collect()
Or
df_x.join(df_y, df_x.id == df_y.id_gigya, 'fullouter').drop(df_x.id).collect()
I got this :
[Row(id=u'_guid_XblBPCaB8qx9SK3D4HuAZwO-1cuBPc1GgfgNUC2PYm4=', countrycode=u'TN', postalcode=u'8160', krux=None),
Row(id=None, countrycode=None, postalcode=None, krux=u'JmJCFu3N'),
Row(id=None, countrycode=None, postalcode=None, krux=u'KNPQLQth'),
Row(id=u'_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', countrycode=u'FR', postalcode=u'75001', krux=u'KlGZj08d')]
[Row(countrycode=u'TN', postalcode=u'8160', id_gigya=None, krux=None),
Row(countrycode=None, postalcode=None, id_gigya=u'_guid_oX6Lu2xxHtA_T93sK6igyW5RaHH1tAsWcF0RpNx_kUQ=', krux=u'JmJCFu3N'),
Row(countrycode=None, postalcode=None, id_gigya=u'_guid_hG88Yt5EUsqT8a06Cy380ga3XHPwaFylNyuvvqDslCw=', krux=u'KNPQLQth'),
Row(countrycode=u'FR', postalcode=u'75001', id_gigya=u'_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', krux=u'KlGZj08d')]
My objectif is to have, anyway, an id by row.. Ideas ? Thx !
Once you have your joined dataset, you can run another select
to output specific columns, then convert to rdd, map it to get only non-null IDs:
df.select('id','id_gigya','countrycode','postalcode')\
.rdd\
.map(lambda x: Row(id=(x.id if x.id_gigya == None else x.id_gigya), postalcode=x.postalcode, countrycode=x.countrycode))\
.collect()
which outputs:
[
Row(countrycode=u'TN', id=u'_guid_XblBPCaB8qx9SK3D4HuAZwO-1cuBPc1GgfgNUC2PYm4=', postalcode=u'8160'),
Row(countrycode=None, id=u'_guid_hG88Yt5EUsqT8a06Cy380ga3XHPwaFylNyuvvqDslCw=', postalcode=None),
Row(countrycode=u'FR', id=u'_guid_YWKnKkcrg_Ej0icb07bhd-mXPjw-FcPi764RRhVrOxE=', postalcode=u'75001'),
Row(countrycode=None, id=u'_guid_oX6Lu2xxHtA_T93sK6igyW5RaHH1tAsWcF0RpNx_kUQ=', postalcode=None)
]
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.