How do I solve "ERROR: missing data for column" with "psql"?

Question

I have a makefile in which I am trying to copy the output of a Python program into a table hosted on my PostgreSQL server.

My query looks something like this:

Python3 filter.py | psql -X -U $(DBUSER) -d $(DBNAME) -h $(DBHOST) -p $(DBPORT) -1 -e \
   -c "COPY table1(col1, col2) FROM STDIN with (format csv, header true, delimiter '|')"

However, when I execute it I get the following error message:

ERROR:  missing data for column "col1"
CONTEXT:  COPY table1, line 168061: ""

To try and understand the issue, I exported the output of the Python program to a CSV file and I realized that the issue comes from the fact that there are no more entries after line 168060. So line 168061 is indeed empty.

I have tried using the NULL option for COPY but it did not work, I get the same error.

I also managed to import into table1 the content of the physical CSV file using a very similar COPY statement. This works, but my objective would be to achieve this without having to create a file.

Any ideas?

Answer 1

解决方案是更改filter.py以使其不输出有问题的空行。

Answer 2

thank you all for your answers,

Indeed I simply did an easy modification in the filter.py and it works now. I was just too focused on doing it in during the import with PSQL.

Cheers :)