Dollar sign in function call in Java using Spark SQL

Question

I have the following code (Java with Spark SQL) -

    import static org.apache.spark.sql.functions.col;
    ...

    System.out.println("=== Filtering records with average age more than 20 ===");
    Dataset<Row> result = df.filter(col("age").$less(20));

I never met a function call in Java that starts with a dollar. Tried googling it but my best guess so far is that it's a result of a Java calling Scala code (but in Scala source code there is no $less function)

Could you please provide a solid explanation for this?

Answer 1

The answer can be found here - http://www.codecommit.com/blog/java/interop-between-java-and-scala

Operators are Methods

One of the most obvious differences between Java and Scala is that Scala supports operator overloading. In fact, Scala supports a variant of operator overloading which is far stronger than anything offered by C++, C# or even Ruby. With very few exceptions, any symbol may be used to define a custom operator. This provides tremendous flexibility in DSLs and even your average, every-day API (such as List and Map ).

Obviously, this particular language feature is not going to translate into Java quite so nicely. Java doesn't support operator overloading of any variety, much less the über-powerful form defined by Scala. Thus, Scala operators must be compiled into an entirely non-symbolic form at the bytecode level, otherwise Java interop would be irreparably broken, and the JVM itself would be unable to swallow the result.

A good starting place for deciding on this translation is the way in which operators are declared in Scala: as methods. Every Scala operator (including unary operators like ! ) is defined as a method within a class:

 abstract class List[+A] { def ::[B >: A](e: B) = ... def +[B >: A](e: B) = ... } 

Since Scala classes become Java classes and Scala methods become Java methods, the most obvious translation would be to take each operator method and produce a corresponding Java method with a heavily-translated name. In fact, this is exactly what Scala does. The above class will compile into the equivalent of this Java code:

 public abstract class List<A> { public <B super A> List<B> $colon$colon(B e) { ... } public <B super A> List<B> $plus(B e) { ... } } 

Every allowable symbol in Scala's method syntax has a corresponding translation of the form “ $trans “. A list of supported translations is one of those pieces of documentation that you would expect to find on the Scala website. However, alas, it is absent. The following is a table of all of the translations of which I am aware:

 ┌────────────────┬─────────────┐ │ Scala Operator │ Compiles To │ ├────────────────┼─────────────┤ │ = │ $eq │ ├────────────────┼─────────────┤ │ > │ $greater │ ├────────────────┼─────────────┤ │ < │ $less │ ├────────────────┼─────────────┤ │ + │ $plus │ ├────────────────┼─────────────┤ │ - │ $minus │ ├────────────────┼─────────────┤ │ * │ $times │ ├────────────────┼─────────────┤ │ / │ div │ ├────────────────┼─────────────┤ │ ! │ $bang │ ├────────────────┼─────────────┤ │ @ │ $at │ ├────────────────┼─────────────┤ │ # │ $hash │ ├────────────────┼─────────────┤ │ % │ $percent │ ├────────────────┼─────────────┤ │ ^ │ $up │ ├────────────────┼─────────────┤ │ & │ $amp │ ├────────────────┼─────────────┤ │ ~ │ $tilde │ ├────────────────┼─────────────┤ │ ? │ $qmark │ ├────────────────┼─────────────┤ │ | │ $bar │ ├────────────────┼─────────────┤ │ \\ │ $bslash │ ├────────────────┼─────────────┤ │ : │ $colon │ └────────────────┴─────────────┘ 

Using this table, you should be able to derive the “real name” of any Scala operator, allowing its use from within Java. Of course, the idea solution would be if Java actually supported operator overloading and could use Scala's operators directly, but somehow I doubt that will happen any time soon.

**** This answer was published by someone but for some reason removed (would love if the orig owner of the answer could republish it)

Answer 2

because every allowable symbol in Scala's method syntax has a corresponding translation of the form “$trans“. so for your question there is a method in scala that called < and the corresponding method in java will be $less

Other operators will be compiled to:

╔═══════════════════╦══════════════╗
║ Scala Operator    ║ Compiles To  ║
╠═══════════════════╬══════════════╣
║=                  ║$eq           ║
║>                  ║$greater      ║
║<                  ║$less         ║
║+                  ║$plus         ║
║-                  ║$minus        ║
║*                  ║$times        ║
║/                  ║$div          ║
║!                  ║$bang         ║
║@                  ║$at           ║
║#                  ║$hash         ║
║%                  ║$percent      ║
║^                  ║$up           ║
║&                  ║$amp          ║
║~                  ║$tilde        ║
║?                  ║$qmark        ║
║║                  ║$bar          ║
║\                  ║$bslash       ║
║:                  ║$colon        ║
╚═══════════════════╩══════════════╝

more information about this can be found here: http://www.codecommit.com/blog/java/interop-between-java-and-scala