When Integrating Pentaho Kettle into a Java Application, Why is the MongoDB output not working?

Question

I am new to Pentaho Kettle and I am using the latest version, 6.0. I have created several simple transformations and jobs in Spoon.

I have a Job that runs a transformation that simply pulls data from CSV files, adds a couple fields to each row, and sends the rows to MongoDB. I also have an error step( Write to Log ) coming off of the MongoDB Output step.

The job and transformation run perfectly in Spoon and all rows appear in MongoDB.

However, when I run the Job from my Java App, everything runs perfectly except when it gets to the MongoDB Output step. There all the rows go to the Write to Log step and there are no errors recorded.

When I take out the Write to Log step there still no errors recorded and no rows are written to MongoDB.

I have a MongoDB driver in my classpath and it was also suggested to put the classes folder from the Kettle root into the my classpath becuase MongoDB is an OSGi plugin. This I have done. However, I am still not getting data into mongodb.

Further research shows that the MongoDB plugin is run in a Apache Karaf implementation. Do I need to be running my Java application as a plugin in a Karaf container along with the MongoDB plugin or is there a simpler way?

Answer 1

Found the solution

I needed to include the VM argument - DKETTLE_PLUGIN_BASE_FOLDERS=C:\\util\\kettle\\data-integration\\system\\karaf\\system\\pentaho where C:\\util\\kettle is the path on my system.

Note: The MongoDB plugin is running on Apache Karaf. It appears that Pentaho is putting their new plugins in this new folder instead of the data-integration\\plugins folder as they move from Apache Felix to Apache Karaf. That's also the reason for the VM argument.

I also needed to ensure that the following jars are in my classpath: metastore, commons-beanutils, commons-digester, commons-logging, commons-vfs, js(rhino), ognl, log4j, junit, guava, commons-codec, scannotation, javassist pentaho-metaverse-api, org.apache.felix.main, pentaho-mongo-utils, pdi-dataservice-server-plugin, javax.servlet-api-3.0.1