使用perl / SQL将文件插入Postgres bytea列

Question

I'm working with a legacy system and need to find a way to insert files into a pre-existing Postgres 8.2 bytea column using Perl.

So far my searching has lead me to believe the following:

1. there is no consensus best approach for this.
2. lo_import looks promising, but I'm apparently too perl-tarded to get it to work.

I was hoping to do something like the following

my $bind1 = "foo"
my $bind2 = "123"
my $file = "/path/to/file.ext"

my $q = q{
      INSERT INTO generic_file_table
        (column_1,
         column_2,
         bytea_column
        ) 
      VALUES
      (?, ?, lo_import(?))
};
my $sth = $dbh->prepare($q);
$sth->execute($bind1, $bind2, $file);
$sth->finish();`

My script works w/o the lo_import/bytea part. But with it I get this error:

DBD::Pg::st execute failed: ERROR: column "contents" is of type bytea but expression is >of type oid at character 176 HINT: You will need to rewrite or cast the expression.

What I think I'm doing wrong is that I'm not passing the actual binary file to the DB properly. I think I'm passing the file path, but not the file itself. If that's true then what I need to figure out is how to open/read the file into a tmp buffer, and then use the buffer for the import.

Or am I way off base here? I'm open to any pointers, or alternative solutions as long as they work with Perl 5.8/DBI/PG 8.2.

Answer 1

Pg offers two ways to store binary files:

• large objects , in the pg_largeobject table, which are referred to by an oid . Often used via the lo extension. May be loaded with lo_import .

• bytea columns in regular tables. Represented as octal escapes like \\000\\001\\002fred\\004 in PostgreSQL 9.0 and below, or as hex escapes by default in Pg 9.1 and above eg \\x0102 . The bytea_output setting lets you select between escape (octal) and hex format in versions that have hex format.

You're trying to use lo_import to load data into a bytea column. That won't work.

What you need to do is send PostgreSQL correctly escaped bytea data. In a supported, current PostgreSQL version you'd just format it as hex, bang a \\x in front, and you'd be done. In your version you'll have to escape it as octal backslash-sequences and (because you're on an old PostgreSQL that doesn't use standard_conforming_strings ) probably have to double the backslashes too.

This mailing list post provides a nice example that will work on your version, and the follow-up message even explains how to fix it to work on less prehistoric PostgreSQL versions too. It shows how to use parameter binding to force bytea quoting.

Basically, you need to read the file data in. You can't just pass the file name as a parameter - how would the database server access the local file and read it? It'd be looking for a path on the server.

Once you've read the data in, you need to escape it as bytea and send that to the server as a parameter.

Update: Like this:

use strict;
use warnings;
use 5.16.3;

use DBI;
use DBD::Pg;
use DBD::Pg qw(:pg_types);
use File::Slurp;

die("Usage: $0 filename") unless defined($ARGV[0]);
die("File $ARGV[0] doesn't exist") unless (-e $ARGV[0]);
my $filename = $ARGV[0];


my $dbh = DBI->connect("dbi:Pg:dbname=regress","","", {AutoCommit=>0});
$dbh->do(q{
        DROP TABLE IF EXISTS byteatest;
        CREATE TABLE byteatest( blah bytea not null );
});
$dbh->commit();

my $filedata = read_file($filename);
my $sth = $dbh->prepare("INSERT INTO byteatest(blah) VALUES (?)");
# Note the need to specify bytea type. Otherwise the text won't be escaped,
# it'll be sent assuming it's text in client_encoding, so NULLs will cause the
# string to be truncated.  If it isn't valid utf-8 you'll get an error. If it
# is, it might not be stored how you want.
#
# So specify {pg_type => DBD::Pg::PG_BYTEA} .
#

$sth->bind_param(1, $filedata, { pg_type => DBD::Pg::PG_BYTEA });
$sth->execute();
undef $filedata;

$dbh->commit();

Answer 2

Thank you to those who helped me out. It took a while to nail this one down. The solution was to open the file and store it. then specifically call out the bind variable that is type bytea. Here is the detailed solution:

.....
##some variables
my datum1 = "foo";
my datum2 = "123";
my file = "/path/to/file.dat";
my $contents;

##open the file and store it
open my $FH, $file or die "Could not open file: $!";
 {
 local $/ = undef;
 $contents = <$FH>;
 };
close $FH;
print "$contents\n";

##preparte SQL
my $q = q{
  INSERT INTO generic_file_table
    (column_1, 
    column_2, 
    bytea_column
    ) 
  VALUES
  (?, ?, ?)
  };
my $sth = $dbh->prepare($q);
##bind variables and specifically set #3 to bytea; then execute.
$sth->bind_param(1,$datum1);
$sth->bind_param(2,$datum2);
$sth->bind_param(3,$contents, { pg_type => DBD::Pg::PG_BYTEA });
$sth->execute();
$sth->finish();