Passing around DB connection objects between multiple map-reduce jobs

Question

Fundamentally, this question is about: Can the same DB connection be used across multiple processes (as different map-reduce jobs are in real different independent processes).

I know that this is a little trivial question but it would be great if somebody can answer this as well: What happens in case if the maximum number of connections to the DB(which is preconfigured on the server hosting the DB) have exhausted and a new process tries to get a new connection? Does it wait for sometime, and if yes, is there a way to set a timeout for this wait period. I am talking in terms of a PostGres DB in this particular case and the language used for talking to the DB is java.

To give you a context of the problem, I have multiple map-reduce jobs (about 40 reducers) running in parallel, each wanting to update a PostGres DB. How do I efficiently manage these DB read/writes from these processes. Note: The DB is hosted on a separate machine independent of where the map reduce job is running.

Connection pooling is one option but it can be very inefficient at times especially for several reads/writes per second.

Answer 1

Can the same DB connection be used across multiple processes

No, not in any sane or reliable way. You could use a broker process, but then you'd be one step away from inventing a connection pool anyway.

What happens in case if the maximum number of connections to the DB(which is preconfigured on the server hosting the DB) have exhausted and a new process tries to get a new connection?

The connection attempt fails with SQLSTATE 53300 too_many_connections . If it waited, the server could exhaust other limits and begin to have issues servicing existing clients.

---

For a problem like this you'd usually use tools like C3P0 or DBCP that do in-JVM pooling, but this won't work when you have multiple JVMs.

What you need to do is to use an external connection pool like PgBouncer or PgPool-II to maintain a set of lightweight connections from your workers. The pooler then has a smaller number of real server connections and shares those between the lightweight connections from clients.

Connection pooling is typically more efficient than not pooling, because it allows you to optimise the number of active PostgreSQL worker processes to the hardware and workload, providing admission control for work.

An alternative is to have a writer process with one or more threads (one connection per thread) that takes finished work from the reduce workers and writes to the DB, so the reduce workers can get on to their next unit of work. You'd need to have a way to tell the reduce workers to wait if the writer got too far behind. There are several Java queueing system implementations that would be suitable for this, or you could use JMS.

See IPC Suggestion for lots of small data

It's also worth optimizing how you write to PostgreSQL as much as possible, using:

• Prepared statements
• A commit_delay
• synchronous_commit = 'off' if you can afford to lose a few transactions if the server crashes
• Batching work into bigger transactions
• COPY or multi-valued INSERT s to insert blocks of data
• Decent hardware with a useful disk subsystem, not some Amazon EC2 instance with awful I/O or a RAID 5 box with 5400rpm disks
• A proper RAID controller with battery backed write-back cache to reduce the cost of fsync(). Most important if you can't do big batches of work or use a commit delay; has less impact if your fsync rate is lower because of batching and group commit.

See:

• http://www.postgresql.org/docs/current/interactive/populate.html
• http://www.depesz.com/index.php/2007/07/05/how-to-insert-data-to-database-as-fast-as-possible/