How to import CSV with missing data in arangodb?

Question

I tried to import CSV file into arangodb using arangoimp. The imported csv file contain missing cell values.

arangoimp --file sample.csv --type csv --collection sample \
  --create-collection true --server.database poc

The above query throws me a warning

'WARNING at position 11: wrong number of JSON values (got 436, expected 446)'

How can I resolve it?

Answer 1

https://github.com/BurntSushi/xsv has a subcommand, fixlengths, to pad/truncate lines. There are umpteen other ways to “fix” CSV files, but in the end, as @jan pointed out, you have to decide what you want. Simply padding or truncating lines may be the wrong thing to do.

Consider, for example, https://github.com/faradayio/scrubcsv , an excellent tool for “fixing” CSV that does not even consider padding/truncating as an option.

One approach is to use a csv2tsv tool to convert the CSV file to a much more manageable TSV file, and then fix it, eg using awk. arangoimp can handle TSV files, so you wouldn't even need to use a tsv2csv tool.

Answer 2

You need to fix the input document.

How should the database resolve the situation? It could assume something like 'null' or another arbitrary value that would be wrong for some cases. Instead of guessing the only reasonable action is taken: The user is notified about the missing value. You find the same behavior in many places. If you pass broken code to a compiler or interpreter you will be notified as well. No compiler will guess your programming intentions. Your code is rejected and you need to fix it.