Bulk insert C# dataset to AWS PostgreSql

Question

I want to bulk insert data table in .net to AWS postgres table in single shot like sqlbulkcopy. I tried copy command but it require some pgadmin_reader acces. Please guide me how can i approach for this problem.

Answer 1

Following up on my comment and your subsequent question, if you do a Npgsql copy from a file, then you need superuser rights. This is because the file will reside on the PostgreSQL server, which one would think very few users (hopefully only admins) have access to or even know about.

However, if you copy from STDIN - standard input, then that is essentially a stream of data that you will then feed.

A very brief primer/example on how to copy from STDIN follows:

using (NpgsqlConnection conn = new NpgsqlConnection(connectionString))
{
    conn.Open();

    using (var writer = conn.BeginBinaryImport("copy my_table from STDIN (FORMAT BINARY)"))
    {
        foreach (object[] fields in myData)
        {
            writer.StartRow();
            writer.Write(fields[0]); // Assumes varchar
            writer.Write(fields[1], NpgsqlTypes.NpgsqlDbType.Integer);
            writer.Write(fields[2], NpgsqlTypes.NpgsqlDbType.Date);
        }

        writer.Complete();
    }
}

Much more on the Npgsql help site.

Answer 2

Try using a library such as PgPartner to accomplish bulk additions very easily. Check out PgPartner: https://www.nuget.org/packages/PgPartner/

https://github.com/SourceKor/PgPartner#bulkadd-npgsqlconnection-extension

Usage:

// Domain Model Objects
var samples = new List<Sample>()
{
    new Sample { Id = Guid.NewGuid(), Name = "Test", ItemSum = 200, ItemAmount = 10 },
    new Sample { Id = Guid.NewGuid(), Name = "Test 2", ItemSum = 400, ItemAmount = 20 },
    new Sample { Id = Guid.NewGuid(), Name = "Test 3", ItemSum = 800, ItemAmount = 30 },
    new Sample { Id = Guid.NewGuid(), Name = "Test 4", ItemSum = 1200, ItemAmount = 40 },
    new Sample { Id = Guid.NewGuid(), Name = "Test 5", ItemSum = 2400, ItemAmount = 50 }
};

// Instantiate and open PostgreSQL database connection
using var conn = new NpgsqlConnection("<connection string>");
conn.Open();

// Execute BulkAdd by passing objects to insert into table, single object mapping, database schema, and database table
conn.BulkAdd(
    samples,
    (mapper, sample) => {
        mapper.Map("id", sample.Id, NpgsqlDbType.Uuid);
        mapper.Map("name", sample.Name, NpgsqlDbType.Text);
        mapper.Map("amount", sample.ItemAmount, NpgsqlDbType.Numeric);
        mapper.Map("sum", sample.ItemSum, NpgsqlDbType.Integer);
    },
    "public",
    "\"Samples\""
);