在关系和非关系数据库之间同步分页

Question

I use mysql as my main database and I sync some data to elasticsearch to make use of features like fuzzy search and aggregations. However, this problem can be applied to and couple of relational and non-relational databases.

When user searches something, I make query to elastic, get ids (primary keys in mysql) and make another query to mysql database, where I filter by ids that were returned from elastic. I use this approach as you often need to load some additional data from relational database, and it would be hell to maintain these relations inside document-based elastic (eg load user with comment).

Problem is, same filters will not be applied to elastic query and mysql query. In above example, what if you need to filter comments by some user param - that filter will be applied to mysql query, but not elastic. If same filters won't be applied, pagination will mismatch - 2nd page in mysql can be 4th in elastic. If I take all of the ids from elastic (no pagination), I am afraid of a long response time and clusters failing + you can't get more than 10K records from elastic without scroll api.

I need a conceptual solution here, not actual query examples. Feel free to suggest totaly different approach altogether. Also, I don't need a perfect pagination match, since mysql will do pagination anyway. If elastic needs to get more records, it's fine, I just don't want to couse too heavy load.

Answer 1

Im afraid there is no general solution for the problem you are explaining . It varies by your response time expectations; size of data etc.

For example,

• If you can ensure that one side of JOIN data will be much lesser - you could change join direction; First do the query on mySQL and then do an id based terms search in ES.

• Consider using database embedded search like postgres depending on how complex your queries are and other features of ES you are leveraging