在 spark-shell 中导入包时出错

Question

Sorry for this question that seems to be asked from time to time. I have read the answers i've seen but as i'm new to scala, i may have not understood them.

I have a build .jar i'd like to use in spark-shell so i launch the shell kike this :

$ spark-shell --driver-memory 1g --jars lsa.jar

Everything seems to be OK :

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/10/15 16:57:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/10/15 16:57:59 WARN Utils: Your hostname, jice-H81M-HD3 resolves to a loopback address: 127.0.1.1; using 192.168.0.11 instead (on interface enp3s0)
17/10/15 16:57:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/10/15 16:58:04 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.0.11:4040
Spark context available as 'sc' (master = local[*], app id = local-1508079480365).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.

But when I try to use it, ihave an error :

scala> import com.cloudera.datascience.lsa._
<console>:23: error: object cloudera is not a member of package com
       import com.cloudera
              ^

I have seen here : Apache Spark not recognising import from external Jar a reference to org.apache.spark.repl.SparkILoop#require() but i can't figure out how to use this.

Do someone can give me a clue ?

Thanks

PS. result of jar -tvf lsa.jar | grep lsa

     0 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/
  2485 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$merge$1$1.class
  2476 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8$$anonfun$10.class
  3926 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1.class
  1307 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$trimLeastFrequent$1.class
  1391 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$7.class
  2250 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$saveDocFreqs$2.class
  1283 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$13.class
  2071 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1$$anonfun$apply$6.class
  1344 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$3.class
  1442 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2$$anonfun$3.class
  1842 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$documentFrequencies$2.class
  1373 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2$$anonfun$apply$3.class
  2816 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2.class
  1261 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$10.class
  1752 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2.class
  2702 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$2.class
  1721 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$2$$anonfun$1.class
  1557 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$5.class
  1351 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$5.class
  1474 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$1.class
  1273 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsForTerm$1.class
  2317 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$6$$anonfun$apply$5.class
  1841 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$documentFrequencies$1.class
 15390 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$.class
  2897 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1.class
  3454 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8.class
  1300 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$11.class
  1668 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1$$anonfun$2.class
  1312 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$saveDocFreqs$1.class
  1877 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$inverseDocumentFrequencies$1.class
  1371 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$12.class
  1152 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$4.class
  1497 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$1.class
  3952 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1.class
  2241 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$plainTextToLemmas$1.class
  1332 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$14.class
  1373 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2$$anonfun$apply$4.class
  1365 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$12.class
  2500 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$plainTextToLemmas$1$$anonfun$apply$4.class
  1416 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2$$anonfun$apply$2.class
 10049 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA.class
  1401 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$11.class
  1491 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsForDoc$1.class
  1726 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$printIdWeights$1.class
  1331 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2$$anonfun$3$$anonfun$apply$1.class
  2141 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8$$anonfun$9.class
  2055 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1$$anonfun$apply$7.class
  2678 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2.class
  2093 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$6.class
  1742 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1$$anonfun$apply$1.class
  1214 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$1.class
  1430 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$2$$anonfun$apply$3.class
  1389 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$termDocumentMatrix$1.class
  1363 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$13.class
  1839 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$1.class
  2678 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2$$anonfun$3.class
  1439 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$merge$1$1$$anonfun$apply$2.class
  1414 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$2.class
  1399 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$readFile$1.class
  1320 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$1.class
  1562 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$6.class
 20269 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$.class
  2759 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$2.class
  1651 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1$$anonfun$9.class
  1616 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1$$anonfun$8.class
  7015 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia.class
  1562 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$7.class
  1422 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$4.class
     0 Sun Oct 15 16:45:50 CEST 2017 META-INF/maven/com.cloudera.datascience/lsa/
  2152 Mon Mar 16 12:55:28 CET 2015 META-INF/maven/com.cloudera.datascience/lsa/pom.xml
   113 Sun Oct 15 16:45:50 CEST 2017 META-INF/maven/com.cloudera.datascience/lsa/pom.properties

Answer 1

使用 --driver-class-path 而不是 --jars，它可以工作