Apache Camel CSV with Header

Question

I have written a simple test app that reads records from a DB and puts the result in a csv file. So far it works fine but the column names ie headers are not put in the csv file. According to the doc it should be put there. I have also tried it without/with streaming and split but the situation is the same.

In the camel unit-tests in line 182 the headers are put there explicitly: https://github.com/apache/camel/blob/master/components/camel-csv/src/test/java/org/apache/camel/dataformat/csv/CsvDataFormatTest.java

How could this very simple problem be solved without the need to iterate over the headers? I also experimented with different settings but all the same. The eg delimiters have been considered I set but the headers not. Thanks for the responses also in advance.

I used Camel 2.16.1 like this:

final CsvDataFormat csvDataFormat = new CsvDataFormat();
csvDataFormat.setHeaderDisabled(false);
[...]
from("direct:TEST").routeId("TEST")
        .setBody(constant("SELECT * FROM MYTABLE"))
        .to("jdbc:myDataSource?readSize=100") // max 100 records
//      .split(simple("${body}"))             // split the list
//      .streaming()                          // not to keep all messages in memory
        .marshal(csvDataFormat)
        .to("file:extract?fileName=TEST.csv");
[...]

EDIT 1

I have also tried to add the headers from the exchange.in. They are there available with the name "CamelJdbcColumnNames" in a HashSet. I added it to the csvDataFormat like this:

    final CsvDataFormat csvDataFormat = new CsvDataFormat();
    csvDataFormat.setHeaderDisabled(false);
    [...]
    from("direct:TEST").routeId("TEST")
            .setBody(constant("SELECT * FROM MYTABLE"))
            .to("jdbc:myDataSource?readSize=100") // max 100 records
            .process(new Processor() {
                public void process(Exchange exchange) throws Exception {
                    headerNames =                       (HashSet)exchange.getIn().getHeader("CamelJdbcColumnNames");
                    System.out.println("#### Process headernames = " + new ArrayList<String>(headerNames).toString());
                     csvDataFormat.setHeader(new ArrayList<String>(headerNames));
                         }
                })
                .marshal(csvDataFormat)//.tracing()
                .to("file:extract?fileName=TEST.csv");

The println() prints the column names but the cvs file generated does not.

EDIT2 I added the header names to the body as proposed in comment 1 like this:

.process(new Processor() {
    public void process(Exchange exchange) throws Exception {

        Set<String> headerNames = (HashSet)exchange.getIn().getHeader("CamelJdbcColumnNames");
        Map<String, String> nameMap = new LinkedHashMap<String, String>();
        for (String name: headerNames){
            nameMap.put(name, name);
        }
        List<Map> listWithHeaders = new ArrayList<Map>();
        listWithHeaders.add(nameMap);

        List<Map> records = exchange.getIn().getBody(List.class);
        listWithHeaders.addAll(records);
        exchange.getIn().setBody(listWithHeaders, List.class);


        System.out.println("#### Process headernames = " + new ArrayList<String>(headerNames).toString());
        csvDataFormat.setHeader(new ArrayList<String>(headerNames));
     }
})

The proposal solved the problem and thank you for that but it means that CsvDataFormat is not really usable. The exchange body after the JDBC query contains an ArrayList from HashMaps containing one record of the table. The key of the HashMap is the name of the column and the value is the value. So setting the config value for the header output in CsvDataFormat should be more than enough to get the headers generated. Do you know a simpler solution or did I miss something in the configuration?

Answer 1

You take the data from a database with JDBC so you need to add the headers yourself first to the message body so its the first row. The resultset from the jdbc is just the data, not including headers.

Answer 2

I have done it by overriding the BindyCsvDataFormat and BindyCsvFactory

public class BindySplittedCsvDataFormat extends BindyCsvDataFormat {

    private boolean marshallingfirslLot = false;

    public BindySplittedCsvDataFormat() {
        super();
    }

    public BindySplittedCsvDataFormat(Class<?> type) {
        super(type);
    }

    @Override
    public void marshal(Exchange exchange, Object body, OutputStream outputStream) throws Exception {
        marshallingfirslLot = new Integer(0).equals(exchange.getProperty("CamelSplitIndex"));
        super.marshal(exchange, body, outputStream);
    }

    @Override
    protected BindyAbstractFactory createModelFactory(FormatFactory formatFactory) throws Exception {
        BindySplittedCsvFactory bindyCsvFactory = new BindySplittedCsvFactory(getClassType(), this);
        bindyCsvFactory.setFormatFactory(formatFactory);
        return bindyCsvFactory;
    }

    protected boolean isMarshallingFirslLot() {
        return marshallingfirslLot;
    }
}


public class BindySplittedCsvFactory extends BindyCsvFactory {

    private BindySplittedCsvDataFormat bindySplittedCsvDataFormat;

    public BindySplittedCsvFactory(Class<?> type, BindySplittedCsvDataFormat bindySplittedCsvDataFormat) throws Exception {
        super(type);
        this.bindySplittedCsvDataFormat = bindySplittedCsvDataFormat;
    }

    @Override
    public boolean getGenerateHeaderColumnNames() {
        return super.getGenerateHeaderColumnNames() && bindySplittedCsvDataFormat.isMarshallingFirslLot();
    }

}

Answer 3

My solution with spring xml (but I'd like to have an option in for extracting also the header on top:

Using spring xml

<multicast stopOnException="true">
    <pipeline>
      <log message="saving table ${headers.tablename} header to ${headers.CamelFileName}..."/>
      <setBody>     
<groovy>request.headers.get('CamelJdbcColumnNames').join(";") + "\n"</groovy>
      </setBody>
      <to uri="file:output"/>
    </pipeline>

    <pipeline>
      <log message="saving table ${headers.tablename} rows to ${headers.CamelFileName}..."/>
      <marshal>
        <csv delimiter=";" headerDisabled="false" useMaps="true"/>
      </marshal>
      <to uri="file:output?fileExist=Append"/>
    </pipeline>
  </multicast>

http://www.redaelli.org/matteo-blog/2019/05/24/exporting-database-tables-to-csv-files-with-apache-camel/