[英]Spark Data frame select nothing
How to retrieve nothing out of a spark dataframe. 如何从Spark数据框中检索任何内容。
I need something like this, 我需要这样的东西
df.where("1" === "2") df.where(“ 1” ===“ 2”)
I needed this so that I can do a left join with another dataframe. 我需要这样做,以便可以与另一个数据框进行左连接。 Basically I am trying to avoid the data skewing while joining two dataframes by splitting the null and not null key columns and joining them separately and then do a union them. 基本上,我试图通过拆分null和not null键列并将其分别连接,然后进行合并来避免在连接两个数据框时出现数据扭曲。
df1 has 300M records out of which 200M records has Null keys. df1具有300M条记录,其中200M条记录具有Null键。 df2 has another 300M records. df2还有300M条记录。
So to join them, I am splitting the df1 containing null and not null keys separately and then join them with df2. 因此,要加入它们,我将分别拆分包含null和not null键的df1,然后将它们与df2合并。 so to join the null key dataframe with df2, I don't need any records from df2. 因此,要将空键数据框与df2联接,我不需要df2的任何记录。
I can just add the columns from df2 to null key df1, but curious to see if we have something like this in spark 我可以将df2中的列添加到空键df1中,但很好奇我们是否在火花中有这样的内容
df.where("1" === "2") df.where(“ 1” ===“ 2”)
As we do in RDBMS SQLs. 就像在RDBMS SQL中一样。
There many different ways, like limit
: 有很多不同的方法,例如limit
:
df.limit(0)
where with Column
: 与Column
:
import org.apache.spark.sql.functions._
df.where(lit(false))
where with String
expression: 在哪里用String
表达式:
df.where("false")
1 = 2
expressed as 1 = 2
表示为
df.where("1 = 2")
or 要么
df.where(lit(1) === lit(2))
would work as well, but are more verbose than required. 也可以,但是比要求的更为冗长。
where
function calls filter
function at the internal level so you can use filter
as where
函数调用filter
在内部功能,所以你可以使用filter
作为
import org.apache.spark.sql.functions._
df.filter(lit(1) === lit(2))
or 要么
import org.apache.spark.sql.functions._
df.filter(expr("1 = 2"))
or 要么
df.filter("1 = 2")
or 要么
df.filter("false")
or 要么
import org.apache.spark.sql.functions._
df.filter(lit(false))
Any expression that would return false in the filter
function would work . 任何在filter
函数中返回false的表达式都可以使用 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.