[英]How to compare two massive spark dataframes on row level and print the difference
I have two very large spark dataframes.我有两个非常大的火花数据框。 I want to compare it on row level and print only the differences我想在行级别比较它并只打印差异
eg:例如:
df1= firstname:abc lastname:xyz company:123
df2= firstname:abc lastname:xyz company:456
expected output- diff= company(df1):123 company(df2):456
预期产出- diff= company(df1):123 company(df2):456
As of me,there is no optimal solution for the problem you have described.就我而言,您所描述的问题没有最佳解决方案。 Because, a difference between dataframes can be found only when you have a column/reference
on which both the dataframes could be joined.因为,只有当您拥有可以连接两个数据框的column/reference
时,才能找到数据框之间的差异。
With that note, one approach will be to use subtract
function find the difference which helps to some extent.有了这个注意,一种方法是使用subtract
函数找到在某种程度上有所帮助的差异。
>>> df_1.show()
+-----+-----+-----+
|fname|lname|cmpny|
+-----+-----+-----+
| abc| xyz| 123|
+-----+-----+-----+
>>> df_2.show()
+-----+-----+-----+
|fname|lname|cmpny|
+-----+-----+-----+
| abc| xyz| 456|
+-----+-----+-----+
>>> df_1.select('*').subtract(df_2.select('*')).show()
+-----+-----+-----+
|fname|lname|cmpny|
+-----+-----+-----+
| abc| xyz| 123|
+-----+-----+-----+
>>> df_2.select('*').subtract(df_1.select('*')).show()
+-----+-----+-----+
|fname|lname|cmpny|
+-----+-----+-----+
| abc| xyz| 456|
+-----+-----+-----+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.