How best to develop the sql to support Search functionality in a web application?

Question

Like many web applications (business) the customer wants a form that will search across each data field. The form could have 15-20 different fields where the user can select/enter/input to be used in sql (stored procedure).

These are quite typical requests by the user that most every application has to deal with.

The issue really at hand is how to provide the user with this type of interface/option AND establish fast SQL access. The above fields could span 15 different tables and respective sql statements (usually abstracted to a stored procedure) will have as many joins. The data always has to be brought back to a grid type view as well as some report format (often excel).

I/we are finding these sql statements are slow and hard to optimize as the user can enter 1 or 15 different search criteria.

How should this be done? Looking for suggestions/ideas as to how existing large applications deal with these requirements.
Does it really come down to trying to optimize the sql within the stored procedure?

thx

Answer 1

No, you need to employ a real search engine technology to make fulltext search have good performance. No SQL predicate (eg LIKE '%pattern%' ) is going to be scalable.

You don't identify which brand of RDBMS you're using, but every major brand of RDBMS has their own fulltext search capability:

• Microsoft SQL Server: Full-Text Search
• Oracle: Oracle Text (formerly ConText)
• MySQL: FULLTEXT index (MyISAM only)
• PostgreSQL: Text-Search data types and index types
• SQLite: Full-Text Search (FTS)
• IBM DB2: Text Search

There are also third-party solutions for indexing text, such as:

• Apache Lucene / Solr
• Sphinx Search
• Xapian