Pentaho Data Integration transformation, loading fields from csv file (Java API)

Question

I am trying to create simple transformation using Kettle Java API.

Just two blocks, one to read from csv file and the other to write it into text file.

Code:

PluginRegistry.addPluginType(SpoonPluginType.getInstance());
        PluginRegistry.addPluginType(StepPluginType.getInstance());
        PluginRegistry.init();

        TransMeta transMeta = new TransMeta();
        transMeta.setName("testTrans");


        String csvStep = "read from file ";
        CsvInputMeta csvInputMeta = new CsvInputMeta();
        csvInputMeta.setDefault();
        csvInputMeta.setFilename(INPUT_FILE);
        csvInputMeta.setDelimiter(";");


        String csvId = PluginRegistry.getInstance().getPluginId(csvInputMeta);
        StepMeta stepMeta = new StepMeta(csvId, csvStep, csvInputMeta);
        transMeta.addStep(stepMeta);


        TextFileOutputMeta textFileOutputMeta = new TextFileOutputMeta();
        textFileOutputMeta.setDefault();
        textFileOutputMeta.setFilename(OUTPUT_FILE);
        textFileOutputMeta.setFileFormat("txt");

        String outPutStep = "Output step";
        String outputId = PluginRegistry.getInstance().getPluginId(textFileOutputMeta);
        StepMeta stepMeta2 = new StepMeta(outputId, outPutStep, textFileOutputMeta);
        transMeta.addStep(stepMeta2);

        transMeta.addTransHop(new TransHopMeta(stepMeta, stepMeta2));
        transMeta.setName("testTrans");

        String xml = transMeta.getXML();
        DataOutputStream dos = new DataOutputStream(new FileOutputStream(new File(trans.xml)));
        dos.write(xml.getBytes("UTF-8"));
        dos.close();

        Trans trans = new Trans(transMeta);
        trans.execute(null);
        trans.waitUntilFinished();

When I run above code the output is:

INFO  18-09 17:32:08,700 - read from file  - Line number : 50000
INFO  18-09 17:32:08,703 - Output step - linenr 50000
INFO  18-09 17:32:09,147 - read from file  - Line number : 100000
INFO  18-09 17:32:09,149 - Output step - linenr 100000
INFO  18-09 17:32:09,491 - read from file  - Line number : 150000
INFO  18-09 17:32:09,492 - Output step - linenr 150000
INFO  18-09 17:32:09,786 - read from file  - Line number : 200000
INFO  18-09 17:32:09,788 - Output step - linenr 200000

and so on. But my csv file actually contains 4 rows thats look like that:

id;val
1;10
2;15
3;20

The problem is transformation "doesn't know" what the fields are. When I exported transformation into xml file, loaded it into Pentaho Spoon and pressed "Get fields" button everything worked correctly (only 3 rows was read).

I know I can just manually create these fields and set them into csvInputMeta but is there a way to do this automatically just like button "Get fields" in Spoon does?

Answer 1

If anyone is curious, I found a solution.

You have to use your own csv reader...

But you can get some help in class CsvInputDialog (its GUI class). There are methods like getCsv and getInfo , those are private so you can't use them directly but you can use them to write your own method. Then as @Dirk said use setInputFields method.

Or you can find some ready csv parser.