psql - read SQL file and output to CSV

Question

I have a SQL file my_query.sql :

select * from my_table

Using psql , I can read in this sql file:

\\i my_query.sql

Or pass it in as an arg:

psql -f my_query.sql

And I can output the results of a query string to a csv:

\\copy (select * from my_table) to 'output.csv' with csv header

Is there a way to combine these so I can output the results of a query from a SQL file to a CSV?

Answer 1

Unfortunately there's no baked-in functionality for this, so you need a little bash-fu to get this to work properly.

CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' my_query.sql | tr '\n' ' ')"

echo "\\copy ($QUERY) to 'out.csv' with CSV HEADER" | $CONN

---

The sed fun removes all semicolons, comment lines, and end of line comments, and tr converts newlines to spaces (as mentioned in a comment by @abelisto):

-- my_query.sql
select *
from my_table
where timestamp < current_date -- only want today's records
limit 10;

becomes:

select * from my_table where timestamp < current_date limit 10

which then gets passed in to the valid psql command:

\copy (select * from my_table where timestamp < current_date) to 'out.csv' with csv header

---

Here's a script:

sql_to_csv.sh

#!/bin/bash
# sql_to_csv.sh

CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' $1 | tr '\n' ' ')"
echo "$QUERY"

echo "\\copy ($QUERY) to '$2' with csv header" | $CONN > /dev/null

./sql_to_csv.sh my_query.sql out.csv

Answer 2

You could do it using a bash script.

dump_query_to_csv.sh:

#!/bin/bash

# Takes an sql query file as an argument and dumps its results
# to a CSV file using psql \copy command.
#
# Usage:
#
#  dump_query_to_csv.sh <sql_query_file> [<csv_output_filesname>]

SQL_FILE=$1
[ -z $SQL_FILE ] && echo "Must supply query file" && exit
shift

OUT_FILE=$1
[ -z $OUT_FILE ] && OUT_FILE="output.csv" # default to "output.csv" if no argument is passed

TMP_TABLE=ttt_temp_table_xx # some table name that will not collide with existing tables

## Build a psql script to do the work
PSQL_SCRIPT=temp.psql

# create a temporary database table using the SQL from the query file
echo "DROP TABLE IF EXISTS $TMP_TABLE;CREATE TABLE $TMP_TABLE AS" > $PSQL_SCRIPT
cat $SQL_FILE >> $PSQL_SCRIPT
echo ";" >> $PSQL_SCRIPT

# copy the temporary table to the output CSV file
echo "\copy (select * from $TMP_TABLE) to '$OUT_FILE' with csv header" >> $PSQL_SCRIPT

# drop the temporary table
echo "DROP TABLE IF EXISTS $TMP_TABLE;" >> temp.sql

## Run psql script using psql
psql my_database < $PSQL_SCRIPT # replace my_database and add user login credentials as necessary

## Remove the psql script
rm $PSQL_SCRIPT

You'll need to edit the psql line in the script to connect to your database. The script could also be enhanced to take the database and account credentials as arguments.

Answer 3

The accepted solution is correct, but I had Windows and had to make it run via a batch (command) file. Posting it here if someone needs that

@echo off

echo 'Reading file %1'
set CONN="C:\Program Files\PostgreSQL\11\bin\psql.exe" -U dbusername -d mydbname
"C:\Program Files\Git\usr\bin\sed.exe" 's/;//g;/^--/ d;s/--.*//g;' %1 | "C:\Program Files\Git\usr\bin\tr.exe" '\n' ' ' > c:\temp\query.txt
set /p QUERY=<c:\temp\query.txt
echo %QUERY%

echo \copy (%QUERY%) to '%2' WITH (FORMAT CSV, HEADER) | %CONN%

Answer 4

我认为最简单的方法是利用shell的变量扩展能力：

psql -U my_user -d my_db -c "COPY ($(cat my_query.sql)) TO STDOUT WITH CSV HEADER" > my_query_results.csv