[英]Using PostgreSQL COPY FROM STDIN
Is it possible to use PostgreSQL's COPY FROM STDIN
statement to load data from CSV file by passing some sort of Reader
or Writer
object the same way it's done in Java?是否可以使用 PostgreSQL 的COPY FROM STDIN
语句通过传递某种Reader
或Writer
器 object 从 CSV 文件加载数据,就像在 Java 中所做的一样? What library should I use?我应该使用什么图书馆? Kotlin example for reference: Kotlin示例供参考:
val cm = CopyManager(conn as BaseConnection)
val total = cm.copyIn("COPY my_table FROM STDIN FORMAT csv", inputStream)
Looking at the two most popular Golang postgres libraries:查看两个最流行的 Golang postgres 库:
The driver supports this via the standard library methods, but you'll need to consume the io.Reader
variable to pass each CSV row through.驱动程序通过标准库方法支持这一点,但您需要使用io.Reader
变量来传递每个 CSV 行。
package main
import (
"bufio"
"context"
"database/sql"
"fmt"
"io"
"log"
"os"
"os/signal"
"github.com/lib/pq"
)
func main() {
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
if err := run(ctx); err != nil {
log.Fatal(err)
}
}
func run(ctx context.Context) error {
// Open CSV file (this could also be os.Stdin, etc)
f, err := os.Open("/tmp/csv")
if err != nil {
return err
}
defer f.Close()
// Open database connection using pq driver
db, err := sql.Open("postgres", "postgres://postgres@localhost:5432/postgres?sslmode=disable")
if err != nil {
return err
}
defer db.Close()
// Execute copy
if err = copyFrom(ctx, db, "my_table", f); err != nil {
return err
}
return nil
}
func copyFrom(ctx context.Context, db *sql.DB, table string, r io.Reader) error {
query := fmt.Sprintf("COPY %s FROM STDIN WITH (FORMAT csv)", pq.QuoteIdentifier(table))
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, query)
if err != nil {
return err
}
sc := bufio.NewScanner(r)
for sc.Scan() {
if _, err = stmt.ExecContext(ctx, sc.Text()); err != nil {
return err
}
}
if err = sc.Err(); err != nil {
return err
}
if _, err = stmt.ExecContext(ctx); err != nil {
return err
}
return tx.Commit()
}
Note笔记
COPY
statements, with at least one still open在处理COPY
语句的底层代码中报告了许多数据竞争,至少有一个仍然开放 The lower level pgconn.PgConn
type has a CopyFrom
method that allows you to pass an arbitrary statement and io.Reader
.较低级别的pgconn.PgConn
类型有一个CopyFrom
方法,允许您传递任意语句和io.Reader
。 If you're connecting via the stdlib db
package, you can still get access to the underlying pgconn.PgConn
as shown below, although there are other ways of handling connections / pools, etc, when using pgx, so it's worth taking a look at those too.如果您通过标准库db
package 进行连接,您仍然可以访问底层pgconn.PgConn
,如下所示,尽管在使用 pgx 时还有其他处理连接/池等的方法,因此值得一看那些也是。
package main
import (
"context"
"database/sql"
"fmt"
"io"
"log"
"os"
"os/signal"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/stdlib"
)
func main() {
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
if err := run(ctx); err != nil {
log.Fatal(err)
}
}
func run(ctx context.Context) error {
// Open CSV file (this could also be os.Stdin, etc)
f, err := os.Open("/tmp/csv")
if err != nil {
return err
}
defer f.Close()
// Open database connection using pgx driver
db, err := sql.Open("pgx", "postgres://postgres@localhost:5432/postgres?sslmode=disable")
if err != nil {
return err
}
defer db.Close()
// Execute copy
if err = copyFrom(ctx, db, "my_table", f); err != nil {
return err
}
return nil
}
func copyFrom(ctx context.Context, db *sql.DB, table string, r io.Reader) error {
query := fmt.Sprintf("COPY %s FROM STDIN WITH (FORMAT csv)", pgx.Identifier{table}.Sanitize())
conn, err := db.Conn(ctx)
if err != nil {
return err
}
defer conn.Close()
return conn.Raw(func(driverConn any) error {
pgConn := driverConn.(*stdlib.Conn).Conn().PgConn()
_, err := pgConn.CopyFrom(ctx, r, query)
return err
})
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.