Generating XML with DB data through dynamically changing XSD

Question

Considering an example for a shop

I have an XSD for a shop, and a number of tables having data relating to inventory, records etc. I have a DB-file specifying where the data related to the XSD lies in the tables. I need to refer this file and XSD to create XML records for multiple shops

My current solution is to generate JPA entities using HyperJAXB through the XSD and read the data to generate the XML's but I need to make code changes every time the DB-file and XSD have changes.

Is it possible to accommodate these changes at runtime whilst using JPA because the DB structure is complex. How can I minimize the effort required to accommodate changes if not doing at runtime.

Answer 1

Ok based off of your current use case I really think using a product like SSIS which specializes in ETL processes could do this the most efficiently. I am fairly certain there is even a way to avoid translation by having SSIS create the xml.

However, if you want to keep doing the process in java I would recommend moving away from JPA due to an object representing the database. This means you will always have coding changes when the database schema gets updated. I would take a step back and leverage more raw SQL over a JdbcTemplate. You can get a fairly generic process by leveraging a few SQL commands:

SHOW tables

then you can get tall results for each table

SELECT * FROM table

This will return a result set which could be turned into XML for each table with a method like...

public static Document toDocument(ResultSet rs) throws ParserConfigurationException, SQLException  {
   DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
   DocumentBuilder builder = factory.newDocumentBuilder();
   Document doc = builder.newDocument();

   Element results = doc.createElement("Results");
   doc.appendChild(results);

   ResultSetMetaData rsmd = rs.getMetaData();
   int colCount = rsmd.getColumnCount();

   while (rs.next()) {
      Element row = doc.createElement("Row");
      results.appendChild(row);

      for (int i = 1; i <= colCount; i++)   {
         String columnName = rsmd.getColumnName(i);
         Object value = rs.getObject(i);

         Element node = doc.createElement(columnName);
         node.appendChild(doc.createTextNode(value.toString()));
         row.appendChild(node);
      }
  }
   return doc;
}

This should let you print every table as XML, if you need it to be hierarchy based there is obviously a bit more work involved, but this would enable you to create a generic process that can export all database tables as xml.

Based off some conversations with my co-workers the best practices we follow to helping to manage database migrations still has a lot of manual work, but here are some of the mitigation steps we take from our practices.

We use schema versioning with either liquibase or flyway to manage schema changes in our database across multiple environments. This can guarantee that regardless of environment the schema will be correct.

We also generate the schema from hibernate and use schema comparison tools (these tend to be database specific) to validate that the hibernate model will be identical to the database model. This operation is done by our Jenkins build server typically to ensure every build is compatible with the desired database schema version.

Hibernate configuration is also set to hibernate.hbm2ddl.auto=validate to ensure compatibility with the database tables on start up.

Our local application uses the Junit framework and creates an instance of hibernate running on derby for database style unit / integration testing. These tests ensure that previous compatibility should not be broken.

When a database is shared between multiple java applications the JPA layer is occasionally shared across all of these applications and multiple groups will be responsible for that particular code base. This practice helps keep all applications in sync with database changes and leveraging the same access patterns. Typically a host of generic named queries are also created to avoid application teams needing to create custom queries.

Unfortunately we have not come across a silver bullet that could automatically evaluate a database schema change and update the application. Currently the process of driving database changes from application JPA changes isn't accepted due to generation process of hibernate not being ideal for every database. As well as a lot of the risks with the hibernate auto update the schema feature which isn't a production grade feature.

Well that turned out to be quite... wordy, but I hope I gave you some insight into your problem.