在多个文件中进行全文搜索的最佳方法

Question

I am developing a module for full text search in about 1,000,000 files (each file is less than 500 KB, and search conditions such as AND OR, should be available to be applied to each file), and there is a chance that I can upload all the files onto Dictionary or List <string> objects(in-memory) when the app is started.

I am considering the following strategies.

1) List <string> (local in-memory) : Parallel.foreach and apply a regex (as I need indexes of search words...) for each string.

2) Open source : Lucene

3) Open source : Elastic search

4) Open source : Yara ( I am aware that it is for detecting malwares. A developer recommended it for me. It would be appreciated if anyone could let me know the details about it. https://github.com/stellarbear/YaraSharp )

5) Redis or DB (This seems slower than 1))

Which one is the fastest? or is there any other strategies?

Answer 1

Your question is very general. Because I don't know all the parts of your case is very difficult to answer. Anyway. I vote for elasticsearch. You will benefit a very wide amount of options for analyzing and discover your text files out of the box. I think from all the solutions you speaking about it's will be the easest way...