将非规范化数据加载到数据库中

Question

I have a database (postgres) with two tables:

CREATE TABLE invoices (
    id bigint,
    some_data varchar
)

CREATE TABLE charges (
    id bigint,
    invoice_id bigint,
    some_data varchar
)

I'm trying to load a csv file with the following format into this database:

invoice_id, invoice_data, charge_id, charge_data

For example, I could have the following lines in my csv file:

1, $10.00, 1, $2.00
1, $10.00, 2, $5.00
1, $10.00, 3, $3.00
2, $2.00,  4, $1.00
2, $2.00,  5, $1.00
3, $11.00, 6, $11.00

This data should correspond to the following records in the database:

SELECT * FROM invoices;
  id | some_data
-----+-------------
  1  | $10.00
  2  | $2.00
  3  | $11.00

SELECT * FROM charges;
  id | invoice_id | some_data
-----+------------+-------------
  1  | 1          | $2.00
  2  | 1          | $5.00
  3  | 1          | $3.00
  4  | 2          | $1.00
  5  | 2          | $1.00
  6  | 3          | $11.00

Is there a 'best practices' for loading this kind of data? At the moment, I am loading this file into an intermediary table and processing it with a php script (bad). It's quite inefficient. Is there a better way? Should I be loading this into an intermediary table and then using a stored procedure to split up the information? Or should I be processing my .csv file directly and splitting this information in some sort of script?

Answer 1

You can use the COPY command to load the data first into a intermediary table whose structure matches the CSV (for example: COPY intermediary_table FROM '/path/to/csv/charges.csv' DELIMITER ',' CSV;), then select the data into each table. The first query would be a SELECT DISTINCT invoice_id, invoice_data INTO invoices FROM intermediary_table, the second query SELECT DISTINCT charge_id, invoice_id, charge_data INTO charges FROM intermediary_table.

BTW, you most likely do not need to use bigint (unless you expect billions of rows).