Complex data-driven web application in Java - Decision on technologies

Question

Dear Stack Overflow Community,

I am a Java programmer in front of a task of building a complex, data-driven, web application (SaaS) and I'm searching for technologies to use. I have already made some decisions and I believe I'm proficient enough to build the appliaction using just the technologies I have decided for (I'm definitely not saying it would be perfect, just that it would be working). However, I'd like to make my task easier and that's why I need your help.

Brief description of the project

Back-end

The application will be heavily data-driven , meaning that everything will be stored in a self-descripting database. This means the database itself will be entirely described with metadata and the application will not know what data it reads and writes. There won't probably be any regular entities (in terms of JPA @Entity) because the application won't know the structure of the data; it will obtain it from the metadata. Only the metadata will have a pre-determined structure. To put it simply, the metadata is the alpha-omega of the application because it will tell the application WHEN and WHAT to display and HOW to display it.

The application will probably utilize stored procedures to perform some low-level tasks on the data, such as automatical auditing, logging and translating to user's language, thus most likely eliminating any possibility to use ORM frameworks because there won't be just simple CRUD operations. Therefore, JDBC seems like my only option (doesn't it?).

Front-end

The UI will be "dumb" in terms that it will not know what data it is displaying (to some extent, of course). It will just know how to display it based on the metadata which it will obtain from the database. All UI controls (like menu items, buttons, etc) will be created based on current application's state and the UI will NOT know what the controls do. This means that clicking a menu item or a button will just send an identifier of associated action to the back-end and the server will decide what to do.

My goals

My main goal is to have the application as lightweight as possible with as least dependencies as possible. Because the application will be very complex, I'd like to avoid any heavy framework(s) because there is a very high probability that I'd need to customize a lot of its functionality.

What I have already decided for

Please object to the following decisions only if you think they're absolutely non-viable for my application, as I have already implemented some core functionality using these technologies:

• Servlets on Tomcat, Guice DI , AOP ( AspectJ )
  I believe all of these technologies are lightweight enough and I don't need to learn J2EE.

• GWT with GIN -jection on the front-end
  Seems like the best option for me because I'm very familiar with Java and Swing and don't want to write any Javascript, PHP or learn a new language. GIN is a little brother of Guice and I will be using the same syntax and principles on both the client and server.

• MSSQL RDBMS
  This is actually a requirement from company management as I'd much rather like to go with an open-source solution. Too bad for me..

• Maven 2
  I think no-one can object to this :)

What I need help with

• DB communication
  I think that ORM is ruled out (is it?) so I need to use JDBC. Do you think Spring JDBC is lightweight and flexible enough for my use? I would often need to "blindly" read data from database, mapping it to some generic entity (because I won't assume any pre-determined structure), and then send the data using some generic DTO to the client along with the metadata telling it what data it is and how to display it. Or do you know any alternatives? Or should I do this myself?

• Client/Server communication
  GWT and its GWT-RPC mechanism seems not very suited for sending the generic data I need. Although I'm convinced that it's doable using GWT-RPC, are there any alternatives? But I definitely want to use GWT.

• Security
  Do you know any security libraries / frameworks that would help me? I'm aware of the existence of Spring-security; do you think it's flexible enough for my use or I'd be better off implementing that myself? Also, is Spring's IoC an integral part of the Spring framework, or would I be able to continue to use Guice?

• Anything else that you think might be useful?

I really appreciate any advice and suggestions because I wouldn't dare to try to make such decisions myself. Please ask me if you need more information.

Thank you in advance! eQui

Answer 1

I think you are over-engineering the solution. Take a look at

http://thedailywtf.com/Articles/Programming-Sucks!-Or-At-Least,-It-Ought-To-.aspx

If everything is driven by the DB, you are going to have immense difficulty making things happen in the UI, and you aren't going to be able to use many of the tools that make UI development easier.

I also suggest you take a look at Spring Roo, if your application is mainly just updating data in a database.

Answer 2

For the backend, i implemented a program which had a similar interaction with the database. the code was database structure oblivious, instead, it read a config file describing the db and could construct complex sql queries based on this information. most of the code is proprietary, but one bit of it got pushed into an open source project called sqlbuilder . may be useful to you on the backend.

Answer 3

I think you're on the right track, with your tool seclection. Your 100% data driven model is going to be hard to maintain. But I understand that's a requirement not an option. Normal source control is going to fail you becuase of the ui application logic all being in the meta-data. You'll need some good test databases and some way to maintain them, such as regularily mysqldump them out and check them in to souce code control to handle all the differences, etc..

You're wise to stay away from various ORM solutions and just use JDBC for this type of app.

Let me give you some warnings about GWT. On the surface it will abstract all the uglyness of html, javascript away and give you clean heirarchy's BUT ...

1) If the abstraction fails you how do you easily debug?

2) Do you want any of your site to be visible to Google or other search engines, if yes GWT is not for you

3) Do you want to use any HTML5 technogies or do you want to be stuck in IE 5 compatability mode?

So... I think you'll be much better off Implementing the UI as simple HTML controls with a small set of jQuery ajax interactions with the server. You can define an input type in your database, your serverlet can generate an input tag and then you have two options you can have some standard event bindings in jquery to tell your server that button1. is clicked, or that select2 has changed, etc. Your server can send back javascript to change the state of the ui - simply load the javascript inside a div so it runs on the client. or 2) You let the input submit the data to the server and do an old school page refesh and the servlet build the next ui screen based on the database.

Building an interface dynamically in HTML from a database is easy and straight foward compared to doing the same in SWING or Windows Forms. You just have to write out a big text string, been doing that since 1999.

That approach is going to be much more lightweight - simpler to debug, understand and modify in the long run than going with the "GWT automatically compiles to unreadable javascript that doesn't run in my browser for some unknown reason" solution.

Answer 4

UI framework and implications for client/server communication

You say that any UI action will triger the backend (and potentially the DB). This mean that UI interraction will be somewhat slow anyway, and more than that will require a round trip to the server.

GWT is especially suited to avoid as much as possible round trips to the server and do all UI work on client side. In this model, only information that will transit from client to server is real data, and not UI metadata. GWT will do the job, but you'll be using a somewhat low level tool, needed for advenced optimisation you'll be unable to perform anyway...

Framework like ZK or Vaadin seems more suited to what you want to do. The client side has nice widgets with a rich UI, but you manipulate the UI from the server side. The framework manage client/server communication for you (no need of REST, RPC or javascript). The main limitation of theses framework is scalability, with all theses chatty round trip. But because your requirement impose that chatty behaviour anyway, you could really benefit from the abstraction they provide, are they are at not cost in your case.

I have tried both GWT and Zk to do some proof of concept for my company. We ended choosing GWT, because of it's hability to be embedded nicelly into any existing UI and to fine tune what you do... In particular avoid as much as possible rountrip to server. But ZK is really easier and faster in term of developmeent hours.

The side effect is that would totally solve your client/server communication concern, leting the framework performing it in an optimized way (Zk is able to intelligently regroup several UI event before sending them to server).

DB and ORM

For DB design, i tend to think that using fine granularity things in DB will make it very very slow. If each widget is one or several rows in the database you'll have to perform many lookup to perform the simpliest thing.

Problem is if your UI is just a little complex with a few dozen of elements (a few button, checkboxes, labels and widgets), compositing a screen will require lot of requests to the DB. Rendering just one page might be very slow and scalability would be very very bad.

I know this because i worked on somewhat generic bug tracking system with similar (but simpler) requirements than yours and we had exactly this problem.

So i would try to describe UI in some templating or XML format. Maybe you'll not show this data to the user, providing it with a nice abstraction, but instead of performning many queries for just one screen, you'll save the whole screen as one blob.

A really dumb and basic implementation of this would be to store HTML/CSS/PNG file in your DB and load it as needed, with user being responsible for making theses HTML file by hand. Of course this would be terrible for the user. That's why you need a nice and fancy editor UI editor that would work on an intermediary format of your own. Another dumb implementation would be some sort of wiki templating. This is not what you need, you need more. But you have the idea, I would seek in that direction...

For the maintenance and debugging too, this would be far easier to the whole UI description to a few file, to understand what is really implemented than to read lot of tabuled data in your prefered SQL editor. Users would have they export/import format to easily version, backup or experiment.

Security

I would say by hand... Because you have a generic UI generated by user it seem likely that the security will be generic too and dependant of database content.

Hope it help...