How to index large number of files contained in a single directory using solr?

Question

I have large number of plain text files in a directory. I want to index them using solr. I used the following command as mentioned in other sources:

java -Dc=test -Drecursive -Dauto -jar post.jar folder

but it indexed 0 files. If I have less files then the following works:

java -Dc=test -Drecursive -Dauto -jar post.jar folder/*

But in case of large number of files I get this error:

Argument list too long .

Note that I am running the command in ubuntu.

Answer 1

The command line has the limit on the number of the arguments, which you could check by executing getconf ARG_MAX

So, number of the files is exceeding this limit and this is the reason why you get this error. You should use xargs to overcome this limit.

Example of usage is following (you need to adapt this to your command):

find . -name \*xml | xargs java -jar post.jar