我们能否以两种不同的格式（即csv和text）在Solr单核数据中建立索引？

Question

I have data in two formates CSV and TEXT.

1) CSV file contains metadata. ie ModifyScore, Size, fileName etc.

2) actual text are in Text folders having files like a.txt, b.txt etc.

Please is it possible to index such data in Solr in a single core through DIH or another possible way?

Answer 1

According to your use case I would proceed with a custom indexing app. Apparently you want to build your Solr document fetching some field from the CSV and some other field( the content) from the TXT .

Using Java for example, it is going to be quite simple : You can use SolrJ, fetch the data from the CSV and TXT, build each Solr Document and then index it.

I would use the DIH if I can move the data in a DB ( even 2 tables are fine, as DIH supports joins). Out of the box, you may be interested in using the script [1] transformer. Using it in combination with your different data sources could work. You need to play a little bit with it as it's not a direct solution to your problem.

[1] https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler#UploadingStructuredDataStoreDatawiththeDataImportHandler-TheScriptTransformer

Answer 2

Just to mention a couple more possibilities:

1. Use DIH to index txt files into collectionA, and use /update handler to ingest csv directly into collectionB, then use Streaming Expressions to merge both into a third collection that is the one you want to keep. The main advantage is everything is in Solr, no external code.

2. Use DIH to index files (or /update to index csv) and write a Update Request Processor that will intercept docs before they are indexed, that looks up the info from the other source, and adds it to the doc.

Answer 3

是的，信息和代码是可能的，如何为来自多个异构数据源的数据建立索引，请参阅为什么tikaEntityProcesor不为以下数据配置文件中的“文本”字段建立索引？