I want to select all records from one dataframe where its value exists/not exists in another dataframe. How to do this using pyspark dataframes?
Question
I have the two pyspark dataframes. I want to select all records from voutdf where its "hash" does not exist in vindf.tx_hash
How to do this using pyspark dataframe.? I tried a semi join but I am ending up with out of memory errors.
voutdf = sqlContext.createDataFrame(voutRDD,["hash", "value","n","pubkey"])
vindf = sqlContext.createDataFrame(vinRDD,["txid", "tx_hash","vout"])
1 answers
solution1
2 ACCPTED 2019-12-04 10:03:48
You can do it with left-anti join:
df = voutdf.join(vindf.withColumnRenamed("tx_hash", "hash"), "hash", 'left_anti')
left-anti join:
It takes all rows from the left dataset that don't have their matching in the right dataset.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
If one dataframe value exists in another dataframe, then get a value from the dataframe
Pandas assign value if value from one dataframe exists in another dataframe
Python Pandas checking for a value if it exists from one DataFrame to another DataFrame
Check if value from one dataframe exists in another dataframe and create column
Check if value from one dataframe exists in another dataframe
Check if value from one dataframe exists in another dataframe, print all pairs of values
Using two dataframes how can I compare a lookup value as a substring in the column in another dataframe to create a new column if the match exists
How do I clean this dataframe. I want to remove "\n" from the table using python without affecting the table
I want to count the elements of a python list that is within a dataframe, and for the output to be a column in the dataframe. How do I do that?
How to check if value exists in another dataframe in pandas?
Related Tags