使用正则表达式计算Spark（Scala）中的Twitter提及数量

Question

I am new to Spark. I want to output the top 2 twitter mentions using this test.txt file:

"I love to dance @Kelsey, especially with you @Kelsey!"

"Can't believe you went to @harvard. Come on man @harvard"

"I love @harvard"

Essentially, multiple mentions in a single tweet only counts once. So the output would be like:

(2, @harvard)

(1, @Kelsey)

So far, my codes looks like the following:

val tweets = sc.textFile("testFile")

val myReg = """(?<=@)([\\w]+)""".r

val mentions = tweets.filter(x => (myReg.pattern.matcher(x).matches))

However, it would not work because x is still a line and it will not match as a result. Is there anyway I can test the word in the line instead of the line itself? Also, how do I check if that mention is redundant in the tweet?

Answer 1

I adjusted your regex a little and you might need to translate it back to spark syntax, but this way you find all mentions and group them. The .toSet is important to remove duplicates, .toLowercase would also make sense there

  val tweets = List("I love to dance @Kelsey, especially with you @Kelsey!",
                "Can't believe you went to @harvard. Come on man @harvard",
                "I love @harvard")


  val myReg = """(@\w+)""".r

  val mentions = tweets.flatMap(x => myReg.findAllIn(x).toSet).groupBy(identity).mapValues(_.length)

  println(mentions)

Answer 2

That works for me, the regexs is more tweeter exact

val myReg = "(^|[^@\\w])@(\\w{1,15})\\b".r

val mentions = tweets.flatMap(x => myReg.findAllIn(x).matchData.map(_.group(0).trim -> 1)).reduceByKey(_ + _)