有没有办法使用scala过滤不包含spark数据帧中某些内容的字段？

Question

Hopefully I'm stupid and this will be easy.

I have a dataframe containing the columns 'url' and 'referrer'.

I want to extract all the referrers that contain the top level domain 'www.mydomain.com' and 'mydomain.co'.

I can use

val filteredDf = unfilteredDf.filter(($"referrer").contains("www.mydomain."))

However, this pulls out the url www.google.co.uk search url that also contains my web domain for some reason. Is there a way, using scala in spark, that I can filter out anything with google in it while keeping the correct results I have?

Thanks

Dean

Answer 1

You can negate predicate using either not or ! so all what's left is to add another condition:

import org.apache.spark.sql.functions.not

df.where($"referrer".contains("www.mydomain.") &&
  not($"referrer".contains("google")))

or separate filter:

df
 .where($"referrer".contains("www.mydomain."))
 .where(!$"referrer".contains("google"))

Answer 2

You may use a Regex . Here you can find a reference for the usage of regex in Scala. And here you can find some hints about how to create a proper regex for URLs.

Thus in your case you will have something like:

val regex = "PUT_YOUR_REGEX_HERE".r // something like (https?|ftp)://www.mydomain.com?(/[^\s]*)? should work
val filteredDf = unfilteredDf.filter(regex.findFirstIn(($"referrer")) match {
    case Some => true
    case None => false
} )

This solution requires a bit of work but is the safest one.