当我只有总文件的子集时，如何应用TF-IDF？

Question

Practical application:

I have several databases that need to be queried from a single search box. Some of them I have direct access to (they're SQL Server / MySQL), others I can only search via an API.

In an ideal world I would inject all of this data into Elasticsearch and use it to determine relevance. Unfortunately I don't have the resources locally to make that run efficiently. Elastic is taking over 400mb of RAM just while idling without adding any actual data or running queries. It looks like most people using Elasticsearch in production are running machines with 32GB - 64GB of RAM. My organization doesn't have access to anything near that powerful available for this project.

So my next idea is to query all the databases and connect to the API's when the user makes a search. Then I need to analyze the results, determine relevance, and return them to the user. I recognize that this is probably a terrible plan in terms of performance. I'm hoping to use memcached to make things more tolerable.

In my research for finding algorithms to determine relevance, I came across tf-idf. I'm looking to apply this to the results I get back from all the databases.

The actual question

My understanding of tf-idf is that after tokenizing every document in the corpus, you perform a term frequency analysis and then multiply it against the inverse document frequency for the words. The inverse document frequency is calculated by dividing the total document count by the the total number of documents with the term.

The problem with this is that if I'm pulling documents from an API, I don't know the true total number of documents in the corpus. I'm only ever pulling a subset, and based on the way those documents are being pulled they're naturally going to all of the terms in them. Can I still apply tf-idf to this by treating the pool of documents returned by these various sources as a single corpus? What's the best way to go about this?

Bonus question

If you have a suggestion for how to accomplish this without hacking together my own search solution or using Elasticsearch I'm all ears...

Answer 1

As you have noticed Elasticsearch is not built to run in memory constrained environments. If you want to use Elasticsearch, but can't set up a dedicated machine, you might consider using a hosted search solution (eg AWS Elasticsearch, Elastic Cloud, Algolia, etc.). Those solutions still cost though!

There are two great alternatives that require a bit more work (but not as much as writing your own search solution). Lucene is the actual Search Engine that Elasticsearch is written on top of. It does still load quite a bit of the underlying data structures into memory, so, depending on the size of the underlying data you want to index, it could still run out of memory. But, you should be able to fit quite a bit more data in a single Lucene index than in an entire Elasticsearch instance.

The other alternative, which I know slightly less about, is Sphinx. It is also a Search Engine. And it also allows you to specify how much memory to allocate for it to use. It stores the rest of the data on disk.