使用Spark和Java将对象保存在Cassandra中

Question

Just started working with spark and cassandra in Java and I am already stuck with saving data to my cassandra database. here is the java bean class that i have

    public class User implements Serializable {
    public User(){}

    public User(String username, String password){
        this.username = username;
        setPassword(password);
    }

    public User(String username, String password, boolean admin){
        this.username = username;
        this.admin = admin;
        setPassword(password);
    }

    private String username;
    public String getUserName(){ return username; }
    public void setUsername(String username){ this.username = username; }

    private String password;
    public String getPassword(){ return password; }
    public void setPassword(String password){ this.password = password; }

    private Boolean admin = false;
    public boolean isAdmin(){ return admin; }
    public void setAdmin(boolean admin){ this.admin = admin; }

    private Calendar dateRegistered = Calendar.getInstance();
    public Calendar getDateRegistered(){ return dateRegistered; }
}

I have a connection with my cassandra database and try to save data the followint way

JavaRDD<User> usersRDD = sparkContext.parallelize(users);
    javaFunctions(usersRDD).writerBuilder("database", "users", mapToRow(User.class)).saveToCassandra();

where users is a list of initiated users. when i excecute this i get the following error.

java.lang.IllegalArgumentException: requirement failed: Columns not found in class com.app.models.User: [username]
at scala.Predef$.require(Predef.scala:233)
at com.datastax.spark.connector.mapper.ReflectionColumnMapper.columnMapForWriting(ReflectionColumnMapper.scala:91)
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$$anon$1.<init>(MappedToGettableDataConverter.scala:27)
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$.apply(MappedToGettableDataConverter.scala:18)
at com.datastax.spark.connector.writer.DefaultRowWriter.<init>(DefaultRowWriter.scala:17)
at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:31)
at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:29)
at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:269)
at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:37)
at com.datastax.spark.connector.japi.RDDJavaFunctions.saveToCassandra(RDDJavaFunctions.java:59)
at com.datastax.spark.connector.japi.RDDAndDStreamCommonJavaFunctions$WriterBuilder.saveToCassandra(RDDAndDStreamCommonJavaFunctions.java:443)
at com.autobot.context.SparkContext.createUsers(SparkContext.java:56)
at com.autobot.context.SparkContext.createUser(SparkContext.java:51)
at com.autobot.user.UserTest.saveUser(UserTest.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

Answer 1

Maybe this exception comes from inconsistent case of username and it's getters/setters? I believe it should be:

private String userName;
public String getUserName(){ return username; }
public void setUserName(String username){ this.username = username; }

Answer 2

I know the above answer is correct but I want to add addtional explanation. The below article will help to understand the role of mapping between Cassandra and Scala/Java properties in Table/Class

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/4_mapper.md

The column-property naming convention is:

Cassandra column name   Scala/Java property name
count                   count
column_1                column1
user_name               userName