Clojure：Spark Graphx的Scala / Java互操作问题

Question

I am trying to use Spark/GraphX using Clojure & Flambo .

Here is the code I ended up with:

In the project.clj file:

(defproject spark-tests "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [yieldbot/flambo "0.5.0"]]
  :main ^:skip-aot spark-tests.core
  :target-path "target/%s"
  :checksum :warn
  :profiles {:dev {:aot [flambo.function]}
             :uberjar {:aot :all}
             :provided {:dependencies
                        [[org.apache.spark/spark-core_2.10 "1.3.0"]
                         [org.apache.spark/spark-core_2.10 "1.2.0"]
                         [org.apache.spark/spark-graphx_2.10 "1.2.0"]]}})

And then my Clojure core.clj file:

(ns spark-tests.core  
  (:require [flambo.conf :as conf]
            [flambo.api :as f]
            [flambo.tuple :as ft])
  (:import (org.apache.spark.graphx Edge)
           (org.apache.spark.graphx.impl GraphImpl)))

(defonce c (-> (conf/spark-conf)
               (conf/master "local")
               (conf/app-name "flame_princess")))

(defonce sc (f/spark-context c))

(def users (f/parallelize sc [(ft/tuple 3 ["rxin" "student"])
                              (ft/tuple 7 ["jgonzal" "postdoc"])
                              (ft/tuple 5 ["franklin" "prof"])]))

(defn edge
  [source dest attr]
  (new Edge (long source) (long dest) attr))

(def relationships (f/parallelize sc [(edge 3 7 "collab")
                                      (edge 5 3 "advisor")]))

(def g (new GraphImpl users relationships))

When I run that code, I am getting the following error:

1. Caused by java.lang.ClassCastException
   Cannot cast org.apache.spark.api.java.JavaRDD to
   scala.reflect.ClassTag

  Class.java: 3258  java.lang.Class/cast
  Reflector.java:  427  clojure.lang.Reflector/boxArg
  Reflector.java:  460  clojure.lang.Reflector/boxArgs

Disclaimer: I have no Scala knowledge.

Then I thought that it may be because Flambo returns a JavaRDD when we use f/parallelize . Then I tried to convert the JavaRDD into a simple RDD as used in the GraphX example:

(def g (new GraphImpl (.rdd users) (.rdd relationships)))

But the I am getting the same error but for the ParallelCollectionRDD class...

From there, I am have idea of what may be causing this. The Java API for the Graph class is here , the Scala API for the same class is here .

What I am not clear about is how to effectively use that class signature in Clojure:

org.apache.spark.graphx.Graph<VD,ED>

(Graph is an abstract class, but I tried using GraphImpl in this example)

What I am trying to do is to re-create that Scala example using Clojure.

Any hints would be highly appreciated!

Answer 1

Finally got it right (I think). Here is the code that appears to be working:

(ns spark-tests.core
  (:require [flambo.conf :as conf]
            [flambo.api :as f]
            [flambo.tuple :as ft])
  (:import (org.apache.spark.graphx Edge
                                    Graph)
           (org.apache.spark.api.java JavaRDD
                                      StorageLevels)
           (scala.reflect ClassTag$)))

(defonce c (-> (conf/spark-conf)
               (conf/master "local")
               (conf/app-name "flame_princess")))

(defonce sc (f/spark-context c))

(def users (f/parallelize sc [(ft/tuple 3 ["rxin" "student"])
                              (ft/tuple 7 ["jgonzal" "postdoc"])
                              (ft/tuple 5 ["franklin" "prof"])]))

(defn edge
  [source dest attr]
  (new Edge (long source) (long dest) attr))

(def relationships (f/parallelize sc [(edge 3 7 "collab")
                                      (edge 5 3 "advisor")
                                      (edge 7 3 "advisor")]))


(def g (Graph/apply (.rdd users)
                    (.rdd relationships)
                    "collab"
                    (StorageLevels/MEMORY_ONLY)
                    (StorageLevels/MEMORY_ONLY)
                    (.apply ClassTag$/MODULE$ clojure.lang.PersistentVector)
                    (.apply ClassTag$/MODULE$ java.lang.String)))

(println (.count (.edges g)))

What this code returns is 3 which seems to be exact. The main issue was that I was not creating the class using Graph/Apply . In fact, it appears that this is the way to create all the objects (looks to be the constructor...). I have no idea why this is what way, but this is probably due to my lack of Scala knowledge. If anybody knows, just tell me why :)

After that I only had to fill-in the gaps for the signature of the apply function.

One thing to note are the last two parameters:

• scala.reflect.ClassTag<VD> evidence$17
• scala.reflect.ClassTag<ED> evidence$18

This is used to instruct Scala of the vertex attribute type ( VD ) and the edge attribute type ( ED ). The type of ED is the type of the object I used as the third parameter of the Edge class. Then the type of VD is the type of the second parameter of the tuple function.