Streamsets Error - Bad File Descriptor

Question

I was attempting to use Streamsets to query an Oracle database and publish the data into Kafka. I downloaded Streamsets' tarball on my Mac and unzipped it into my home directory. Running $HOME/streamsets-datacollector-2.1.0.2/bin/streamsets dc started up on my first try, then I followed the instructions here to add the jdbc driver, then the instructions here to configure my streamsets job. However, I got an error: JDBC_00 - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: java.sql.SQLRecoverableException: IO Error: Bad file descriptor .

Answer 1

This wound up having something to do with the limit on the number of files a process can have open. When I ran ulimit -n on the laptop, it showed 4864 , then I set it to 10,000 via ulimit -n 10000 , restarted the streamsets server, and it worked! If I need to keep running this, I will find a more procedural way of setting the ulimit for this process to work around this issue.

Answer 2

you definitely want need increase ulimit -n. To permanently change for all users in Ubuntu try the following:

echo "* soft nofile 60000" > /etc/security/limits.d/*_limits.conf && echo "* hard nofile 60000" >> /etc/security/limits.d/*_limits.conf

substitute 60000 with whatever number you want. I've never had a problem in streamsets using 60k. you should be able to do something similar in bsd.