Best way to check if external data is already in our MySQL database

Question

We are looking at the best way of developing a search routine in a database that uses the least resources possible within a PHP/MySQL environment.

We process an external feed of information that has a tendency to subtly change or have new status values from time to time. This means we are limited to add our own numeric keys and search by conventional SQL search.

We are thinking of using MD5 to create a unique string so instead of searching for...

WHERE DATE = '12/12/2012 09:00' 
AND TYPE = 'new alert' 
AND loc = 'rear door' 
AND subtype = 'pir hit' 
AND lat = 39.3343 
AND lon = 145.234 
AND current STATUS = 'active' 
AND Support = 'en-route';

we create a MD5 eg ef6d3c25ac9362413fed2b4d3f65962a out of the fields that we are interested in eg 12/12/2012 09:00~new alert~rear door~pir hit~-39.3343~145.234~active~en-route and then we can search for just this MD5 in the database rather than separate fields.

We would also be interested in looking into using a file with the list of most recent MD5's rather that interrogating the db constantly as we may have 1100+ jobs in the feed at the most. More often it is around 60 or so jobs.

We are interested in your thoughts and reasons on what you think is the best solution.

Answer 1

I'd go for Apache Solr as a solution. Faceted search would suite all your needs here. Replicating/indexing your data wouldn't take much effort. We've implemented this engine in our company's project, doing search over name/date/characteristics/vendor/distributor/etc, and it works like a charm. Though md5 over glued string has been a solution for some time there. Anyways, it depends on time you have and how well your current solution handles the situation.