Django upload csv file copy to postgresql table

Question

I am uploading a csv file in Django :

 $('#import-file-dialog').on('change', function(event) { $('#upload-files').click(); }); 

 <div class="" style="padding-left: 15px;"> <button class="btn btn-primary btn-outline" id="open-file-dialog">Upload Bulk</button> <input type="file" id="import-file-dialog" name="import_file_dialog" style="display: none;" /> <button class="btn btn-primary" id="upload-files" name="upload_import" style="display: none;">Upload Bulk</button> </div> 

Now i want to use psycopg2 copy_from for inserting csv data to Posgresql. In django model:

fa = StringIO(import_data.read().decode("utf-8"))
cursor.copy_from(fa , table_name, sep=',')
commit_changes()

csv file data is getting store with csv header name in my table. How i remove csv header so that i could store that data to table.

Thanks for your precious time.

Answer 1

You are doing double-processing of the CSV file. csv.reader reads the file line-by-line and parses it into lists of fields. copy_from reads a file or other object and parses into fields (default to tab, but with sep=',' it is effectively CSV). You can either:

a) read the file directly (ie, just open it normally instead of using csv ) and then use copy_from or b) use csv.reader to parse and then build queries to insert each row.

The first option is much simpler. But keep in mind (from the cursor class docs ) that there is an optional columns parameter. If you don't use it then The length and types should match the content of the file to read. If not specified, it is assumed that the entire table matches the file structure.

Unfortunately, there is no "skip the header line" option for copy_from . If you need to do that, you can either (if it is a one-time load) delete the line manually, or you can use copy_export - something like:

cursor.copy_expert(sql="COPY %s FROM STDIN WITH CSV HEADER DELIMITER AS ','" % table_name, file=fa)

Full example at Load a CSV File with Header in Postgres via Psycopg

Answer 2

For skipping your CSV file header and copy the data respectively, the code is pretty simple:

cursor = conn.cursor()

with open(csv_file, 'r') as fp:
    next(f)  # Skip header
    cursor.copy_from(fp, table_name, sep=',', columns=columns)

PS Don't forget to pass a tale name and an ordered set of columns, that should exactly being matched to CSV file columns ordering