inserting millions of xml files into basex

Question

Is it possible to store a million of records(xml files) into Basex database?

for(int j=1;j<1000000;j++) {
     String id=String.valueOf(j);
     String[] elems={"firstname","lastname","nickname","salary","permanentAddress","currentAddress","contactNo","email","alternateMail","FacebookName","Color"};
     String[] childs = {id,"yong"+id,"mook kim"+id,"mkyong"+id,"100000"+id,"Bhopal"+id,"pune"+id,"999999999"+id ,"test"+id+"@basex.com","testA"+id+"@basex.com","Tom"+id,"grey"+id};

    try {
        xmlFile= x.CreateXMLDoc("test","transperfect",elems,childs);
    } catch (TransformerConfigurationException exception) { 
        exception.printStackTrace(); 
    }

    storeIntoBaseXDB(j,id,xmlFile);         
}

//StoreIntoBaseXDB method

public static void storeIntoBaseXDB(int i,String id,String xmlFile) throws BaseXException {

    if(i==1)
    {   
        System.out.println("=========================Store into Database=========================");


        //System.out.println("Check database existence");
        con.getDatabaseConnection(dbNmae);
        con.executeInsertQuery(dbNmae,id,xmlFile);
    }
    else
    {
        con.executeInsertQuery(dbNmae,id,xmlFile);

    }
}

//executeInsertQuery() method

          public long executeInsertQuery(String dbname,String id,String xmlFile) throws BaseXException
    {

           //System.out.println(path);
            new Open(dbname).execute(context);



            try 
            {


                /* String query = "db:add('db','C:/Users/manish/Desktop/BaseX65/xml/books.xml')";
                 QueryProcessor proc = new QueryProcessor(query, context);
                    proc.execute();
                    proc.close();
                    context.close();*/

                new Add(id,xmlFile).execute(context);

                //System.out.println(new Find(id));


            }

            catch (Exception e)
            {
                System.out.println("ERROR executing query: ");
                e.printStackTrace();
            } 

}

Answer 1

Yes, it is possible. Why wouldn't it be (a million is a relatively small number)?

The limit is 2^29 which is 546,870,912 (the number of stored XML documents).

The limit for XML elements is 2^31 which is 2,147,483,648 (although this includes all nodes including attributes, texts, etc.).

Here is a web page on BaseX statistics regarding example existing databases, the first row of the table also contains the limits ( #Files represents the number of stored XML documents, #Nodes represents the number of XML nodes like elements, attributes, texts, etc.):

http://docs.basex.org/wiki/Statistics

Answer 2

Adding to the answer @icza provided with the correct statistics for BaseX, I'd like to add a few things:

The upper limits on number of files, nodes, etc. pp. always apply per database . This means you can simply create another database to circumvent these restrictions. In many situations it might even be beneficial to split a large database up into several small ones, as locking is done on a database level, ie if you have one large database and you do a write, everything is locked. If you split it up into several databases, only a smaller portion will be locked.

As XQuery can easily access multiple databases in one query, split-up databases can easily be merged together in your queries.