How do I resolve a Room invalid schema error involving 'notNull' and 'primaryKeyPosition' on prepopulated database?

Question

My Problem : I'm converting an existing Java Desktop app and I'm learning Android Room development and have a prepopulated SQLite database created through SQLite Studio that contains a many-to-many join table (ie Author_By_Source ). This database and table are used in an existing Java desktop app. I'm trying to resolve the invalid schema mismatching between the ' notNull ' and ' primaryKeyPosition ' properties. I have not been able to get these two properties to update. This is the segment of the error I'm trying to resolve for the AuthorID field, but it is also the same differences for the SourceID in the same error message. So both fields need the same resolution:

    Expected:
    TableInfo{name='Author_By_Source', columns={AuthorID=Column{name='AuthorID', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='null'}, ...
    Found:
    TableInfo{name='Author_By_Source', columns={AuthorID=Column{name='AuthorID', type='INTEGER', affinity='3', notNull=false, primaryKeyPosition=0, defaultValue='null'}, ...

What I have tried : I have tried using the SQLite Studio interface used to build and structure the database and edited the table structure for the ' Author_By_Source ' table. I set each field to Not Null but it is not reflected as 'true' when I rerun the app. I also haven't identified how to designate the primaryKeyPosition. I've tried the reverse by reading more thoroughly the options and methods of the Room Annotations but I haven't been successful. I have been reading through some Room documentation throughout my project and visited some forums and tutorials, but not found anything addressing quite this specific.

What I'm trying to do : I'm trying to use a prepopulated database with multiple tables of which some have join tables for many-to-many relationships. I am hoping for some direction.

Below is how I annotated my AuthorBySource class which may be of assistance:

 @Entity(tableName = "Author_By_Source", primaryKeys = {"AuthorID", "SourceID"},  foreignKeys = {
        @ForeignKey(entity = Authors.class, parentColumns = "AuthorID", childColumns = "AuthorID"),
        @ForeignKey(entity = Sources.class, parentColumns = "SourceID", childColumns = "SourceID")},
        indices = {@Index("AuthorID"), @Index("SourceID")})
public class AuthorBySource {
    @ColumnInfo(name = "AuthorID")
    private int authorID;
    @ColumnInfo(name = "SourceID")
    private int sourceID;

Answer 1

Are you re-copying the asset file and uninstalling the App each time you makes changes? If not and this doesn;t resolve the issue then :-

If you compile the room code then look at java(generated) in your_database_class _impl (where your_database_class is your @Database class name) and at the method createAllTables . This has SQL used to create the database (you can ignore the room_master SQL).

For example (this based upon your Entity and minimal referenced tables) :-

  @Override
  public void createAllTables(SupportSQLiteDatabase _db) {
    _db.execSQL("CREATE TABLE IF NOT EXISTS `Authors` (`AuthorID` INTEGER, `AuthorName` TEXT, PRIMARY KEY(`AuthorID`))");
    _db.execSQL("CREATE TABLE IF NOT EXISTS `Sources` (`SourceID` INTEGER, `SourceName` TEXT, PRIMARY KEY(`SourceID`))");
    _db.execSQL("CREATE TABLE IF NOT EXISTS `Author_By_Source` (`AuthorID` INTEGER NOT NULL, `SourceID` INTEGER NOT NULL, PRIMARY KEY(`AuthorID`, `SourceID`), FOREIGN KEY(`AuthorID`) REFERENCES `Authors`(`AuthorID`) ON UPDATE NO ACTION ON DELETE NO ACTION , FOREIGN KEY(`SourceID`) REFERENCES `Sources`(`SourceID`) ON UPDATE NO ACTION ON DELETE NO ACTION )");
    _db.execSQL("CREATE INDEX IF NOT EXISTS `index_Author_By_Source_AuthorID` ON `Author_By_Source` (`AuthorID`)");
    _db.execSQL("CREATE INDEX IF NOT EXISTS `index_Author_By_Source_SourceID` ON `Author_By_Source` (`SourceID`)");
    _db.execSQL("CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)");
    _db.execSQL("INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '953f8512db886fb7d206fa561a7117c4')");
  }

You need to convert the database accordingly, that is to be as per the definition, so for example you could for the above, in SQLite Studio, use

/* Just once */
PRAGMA foreign_keys = 0;

/* Per table */
DROP TABLE IF EXISTS Author_By_Source_new;
DROP TABLE IF EXISTS Author_By_Source_original;
CREATE TABLE IF NOT EXISTS Author_By_Source_new copy_generated_sql_for_the_column_definitions_etc;
DELETE FROM Author_By_Source_new;
INSERT INTO Author_By_Source_new SELECT * FROM Author_By_Source;
ALTER TABLE Author_By_Source RENAME TO Author_By_Source_original;
ALTER TABLE Author_By_Source_new RENAME TO Author_By_Source;
/* IF HAPPY THEN DO */
-- DROP TABLE IF EXISTS Author_By_Source_original; /* NOTE TURNED OFF */

...... do the equivalent for all tables


/* Just once */
PRAGMA foreign_keys = 1;
PRAGMA foreign_key_check;
PRAGMA integrity_check; 

• copy_generated_sql_for_the_column_definitions_etc should be replaced accordingly by the column definition part from the generated SQL (or copy the entire statement and change the table name).
• Note that the INSERT INTO .... statement assumes that the column order is the same for the new and original tables, otherwise use INSERT INTO (column_list).... where the columns are as per the generated SQL eg INSERT INTO Author_By_Source_new (AuthorID,SourceID) SELECT * FROM Author_By_Source;

• The CREATE TABLE SQL would be (based upon the generated SQL above)

  • CREATE TABLE IF NOT EXISTS `Author_By_Source_new` (`AuthorID` INTEGER NOT NULL, `SourceID` INTEGER NOT NULL, PRIMARY KEY(`AuthorID`, `SourceID`), FOREIGN KEY(`AuthorID`) REFERENCES `Authors`(`AuthorID`) ON UPDATE NO ACTION ON DELETE NO ACTION , FOREIGN KEY(`SourceID`) REFERENCES `Sources`(`SourceID`) ON UPDATE NO ACTION ON DELETE NO ACTION );
  • NOTE the table name has been changed ie suffixed with _new

Example

Here's an Example based upon what's available from your code.

The Original pre-populated database

A database was generated using an SQLite Management Tool (Navicat) using :-

DROP TABLE IF EXISTS Author_By_Source;
DROP TABLE IF EXISTS Authors;
DROP TABLE IF EXISTS Sources;
CREATE TABLE IF NOT EXISTS Authors (AuthorID INTEGER PRIMARY KEY, AuthorName TEXT);
CREATE TABLE IF NOT EXISTS Sources (SourceID INTEGER PRIMARY KEY, SourceName TEXT);
/* Create the table so that it would cause issues in Room  */
CREATE TABLE IF NOT EXISTS Author_By_Source (
    AuthorId INTEGER, SourceId INTEGER, 
    FOREIGN KEY (AuthorID) REFERENCES Authors(AuthorID), 
    FOREIGN KEY (SourceID) REFERENCES Sources(SourceID)
);

INSERT INTO Authors (AuthorName) VALUES ('Fred'),('Mary'),('Joan'),('Bert'),('Alan');
INSERT INTO Sources (SourceName) VALUES ('S1'),('S2'),('S3'),('S4'),('S5');
INSERT INTO Author_By_Source VALUES(1,1),(2,2),(3,3),(4,4),(5,5),(1,4),(3,4),(5,2);

SELECT AuthorName, SourceName 
    FROM Author_By_Source 
        JOIN Authors ON Author_By_Source.AuthorID = Authors.AuthorID
        JOIN Sources ON Author_By_Source.SourceID = Sources.SourceID
;

After running it being :-

AuthorName  SourceName
Fred        S1
Mary        S2
Joan        S3
Bert        S4
Alan        S5
Fred        S4
Joan        S4
Alan        S2

As expected trying to use this results in :-

java.lang.RuntimeException: Unable to start activity ComponentInfo{a.a.so59756782javaroomprepopulatedconversion/a.a.so59756782javaroomprepopulatedconversion.MainActivity}: java.lang.IllegalStateException: Pre-packaged database has an invalid schema: Authors(a.a.so59756782javaroomprepopulatedconversion.Authors).
 Expected:
TableInfo{name='Authors', columns={AuthorName=Column{name='AuthorName', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='null'}, AuthorID=Column{name='AuthorID', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='null'}}, foreignKeys=[], indices=[]}
 Found:
TableInfo{name='Authors', columns={AuthorID=Column{name='AuthorID', type='INTEGER', affinity='3', notNull=false, primaryKeyPosition=1, defaultValue='null'}, AuthorName=Column{name='AuthorName', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='null'}}, foreignKeys=[], indices=[]}

Converting

The following, based upon the answer, was used to convert the database :-

BEGIN TRANSACTION;
DROP TABLE IF EXISTS Author_By_Source;
DROP TABLE IF EXISTS Authors;
DROP TABLE IF EXISTS Sources;
CREATE TABLE IF NOT EXISTS `Authors` (`AuthorID` INTEGER NOT NULL, `AuthorName` TEXT, PRIMARY KEY(`AuthorID`));
CREATE TABLE IF NOT EXISTS `Sources` (`SourceID` INTEGER NOT NULL, `SourceName` TEXT, PRIMARY KEY(`SourceID`));
/* Create the table so that it would cause issues in Room  */
CREATE TABLE IF NOT EXISTS Author_By_Source (
    AuthorId INTEGER, SourceId INTEGER, 
    FOREIGN KEY (AuthorID) REFERENCES Authors(AuthorID), 
    FOREIGN KEY (SourceID) REFERENCES Sources(SourceID)
);

INSERT INTO Authors (AuthorName) VALUES ('Fred'),('Mary'),('Joan'),('Bert'),('Alan');
INSERT INTO Sources (SourceName) VALUES ('S1'),('S2'),('S3'),('S4'),('S5');
INSERT INTO Author_By_Source VALUES(1,1),(2,2),(3,3),(4,4),(5,5),(1,4),(3,4),(5,2);

SELECT AuthorName, SourceName 
    FROM Author_By_Source 
        JOIN Authors ON Author_By_Source.AuthorID = Authors.AuthorID
        JOIN Sources ON Author_By_Source.SourceID = Sources.SourceID
;   

/* Just once */
PRAGMA foreign_keys = 0;

/* Per table */
DROP TABLE IF EXISTS Author_By_Source_new;
DROP TABLE IF EXISTS Author_By_Source_original;
CREATE TABLE IF NOT EXISTS `Author_By_Source_new` 
    (
        `AuthorID` INTEGER NOT NULL, 
        `SourceID` INTEGER NOT NULL, 
        PRIMARY KEY(`AuthorID`, `SourceID`), 
        FOREIGN KEY(`AuthorID`) REFERENCES `Authors`(`AuthorID`) ON UPDATE NO ACTION ON DELETE NO ACTION , 
        FOREIGN KEY(`SourceID`) REFERENCES `Sources`(`SourceID`) ON UPDATE NO ACTION ON DELETE NO ACTION 
    );
DELETE FROM Author_By_Source_new;
INSERT INTO Author_By_Source_new SELECT * FROM Author_By_Source;
ALTER TABLE Author_By_Source RENAME TO Author_By_Source_original;
ALTER TABLE Author_By_Source_new RENAME TO Author_By_Source;
/* IF HAPPY THEN DO */
DROP TABLE IF EXISTS Author_By_Source_original;

-- ...... do the equivalent for all tables

CREATE INDEX IF NOT EXISTS `index_Author_By_Source_SourceID` ON `Author_By_Source` (`SourceID`);
CREATE INDEX IF NOT EXISTS `index_Author_By_Source_AuthorID` ON `Author_By_Source` (`AuthorID`);

COMMIT;

/* Just once */
PRAGMA foreign_keys = 1;
PRAGMA foreign_key_check;
PRAGMA integrity_check;
SELECT sql FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite%';

SELECT AuthorName, SourceName 
    FROM Author_By_Source 
        JOIN Authors ON Author_By_Source.AuthorID = Authors.AuthorID
        JOIN Sources ON Author_By_Source.SourceID = Sources.SourceID
;

The database was closed, timestamps and size checked, and then database file replaced the existing file in the assets folder. The App was uninstalled and then rerun with the following in the MainActivity :-

    myDB = Room.databaseBuilder(this,MyAppDatabase.class,"mydb")
            .createFromAsset("mydb")
            .allowMainThreadQueries()
            .build();
    myDB.getOpenHelper().getWritableDatabase();

    List<AuthorWithSources> authorWithSourcesList = myDB.allDao().getAllAuthorsWithSources();
    for (AuthorWithSources aws: authorWithSourcesList) {
        Log.d("AUTHORSOURCEINFO","Author = " + aws.authors.getAuthorName() + "Source = " + aws.sources.getSourceName());
    }
}

• MyAppDatabase being the @Database class.
• allowMainThreadQueries used for convenience.

• myDB.getOpenHelper().getWritableDatabase(); used to force an immediate open and therefore copy of the database.

• getAllAuthorsWithSources() does as it says using @Query("SELECT * FROM Author_By_Source") List<AuthorWithSources> getAllAuthorsWithSources();

AuthorWithSources being :-

public class AuthorWithSources {

    @Embedded
    AuthorBySource authorBySource;
    @Relation(entity = Authors.class,parentColumn = "AuthorID",entityColumn = "AuthorID")
    Authors authors;
    @Relation(entity = Sources.class,parentColumn = "SourceID",entityColumn = "SourceID")
    Sources sources;
}

Result

No Errors and in the log :-

 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = FredSource = S1 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = MarySource = S2 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = JoanSource = S3 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = BertSource = S4 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = AlanSource = S5 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = FredSource = S4 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = JoanSource = S4 2020-01-16 10:09:23.623 D/AUTHORSOURCEINFO: Author = AlanSource = S2

Alternative Approach

A tool, as an App, exists that converts databases for use by Room and generates the java code for the Entities/ Dao's.

The tool is detailed here RoomExistingSQLiteDBConverter

Using the tool and

• copying the generated java source code into the project and then
• visiting each file a resolving the imports (assumes that the correct package name was input otherwise the package would also have to be adjusted or added) and then
• creating the assets folder in the project and then
• copying the generated database

• and finally creating code to use the database eg

  soanswersDatabase = Room.databaseBuilder(this,SoanswersDatabase.class,SoanswersDatabase.DBNAME) .allowMainThreadQueries() .createFromAsset(SoanswersDatabase.DBNAME) .build(); List<Author_By_Source> authorBySourceList = soanswersDatabase.getAuthor_By_SourceDao().getEveryAuthor_By_Source(); for (Author_By_Source abs: authorBySourceList) { Log.d("AUTHORSOURCEINFO", "AuthorReference = " + String.valueOf(abs.getAuthorId()) + " Sourcereference = " + String.valueOf(abs.getSourceId()) ); }

• ie the above is the only code that was typed in. Results in :-

 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 1 Sourcereference = 1 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 2 Sourcereference = 2 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 3 Sourcereference = 3 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 4 Sourcereference = 4 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 5 Sourcereference = 5 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 1 Sourcereference = 4 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 3 Sourcereference = 4 2020-01-16 10:46:27.907 D/AUTHORSOURCEINFO: AuthorReference = 5 Sourcereference = 2

Answer 2

I was able to resolve this issue but what it amounted to was matching the external database field constraints to the Android Room Entities and learning a bit more about the Room Annotations. The issues were basically whether the field was notNull. For some reason, which could be a naive misunderstanding on my part, is when creating Entities with Android Room, fields would seem to automatically default to 'notNull' which I personally wouldn't expect unless I specified. However, through the experience I think I have a more stable database than I would have.