处理 SQL 查询的自然语言

Question

I work in C# (Entity Framework) and Postgresql, but I'm not opposed to using Python or even Javascript. I want to be able to process searches that produce relevant results. For example, let's say I have a row in a very large database where display_name is Mike's® Discount Auto , and I want users to be able to search for it using a variety of ways. I've been using LINQ and Levenshtein distance stuff, but I can't seem to get it quite right. For the example above, I want the following searches to actually find Mike's® Discount Auto

1. Mike's
2. Mikes
3. Mike's®
4. Mikes®
5. Miikes
6. Mikes discount auto
7. discount auto

yada yada yada. Each of my strategies seems to work ok , but there are huge gaps. I use regex to remove non-alphanumeric characters, I use Levenshtein distance to search for misspellings, but even those 2 strategies won't effectively work if someone types in Mikes , because the Levenshtein distance is very high compared to something like Bobs discount auto . For the second example, the distance is lower, but obviously not the correct one. Plus the more things I add, the slower the search becomes. Right now with a database consisting of ~330,000 rows, it takes almost a full minute from the http request -> lambda -> database -> back to client. That's not acceptable, My lambda most definitely needs to be faster. but it's my code that is really slowing it down

I'm looking for any resource in how to handle this effectively (ie books, websites, courses on udemy).

Answer 1

Have you tried pg_trgm ? It is not particularly intelligent (it doesn't understand parts of speech or synonyms or conjugation, for example) but neither is Levenshtein, and pg_trgm is usually much much faster as it can use an index.

And neither one will have problems identifying "Mikes" as being more similar to 'Mikes Discount Auto' than to 'Bobs discount auto', so your example doesn't make sense.