如何并行运行一些作业

Question

After reading all posts about parallelism in Ruby, I got only confused, so I will describe what I want to do.

I have names that contains around 1000 names.

names
=> [{"name"=>"tickets"}, {"name"=>"events"}, {"name"=>"channel"}, {"name"=>"primes"}]

For each name I want to drop a table if it exists using pg.

drop_str = "DROP TABLE IF EXISTS %s ;"
create_str = "CREATE TABLE %s (id SERIAL PRIMARY KEY,bkk varchar(255))"

names.each do |name|
    conn.exec((drop_str % name["name"]) + (create_str % name["name"]))
end

But, I do not want to drop tables one after another one. I want to do it in parallel.

My idea is to use following:

threads = []
drop_str = "DROP TABLE IF EXISTS %s ;"
create_str = "CREATE TABLE %s (id SERIAL PRIMARY KEY,bkk varchar(255))"

names.each do |name|
    threads.push(Thread.new{conn.exec((drop_str % name["name"]) + (create_str % name["name"]))})
end

and then to join the threads.

In reality will the tables be dropped in parallel or still one after another one?

Answer 1

In principle, you can run multiple SQL statements in parallel. Most database engines are multi-threaded and can execute multiple statements in parallel Sometimes it doesn't make much sense though, as when using SQLite.

There are several caveats though which will probably break your current code.

Most importantly, a single connection to your database always has some state attached to it. Often, it will hold transactions and internal state of the database adapter. As such, a single database connection is generally only usable in a single thread at a time. If you attempt to send multiple parallel statements over a single connection, things will probably break pretty in-deterministically.

Thus when trying to run multiple statements in parallel using threads, each threads needs its own database connection. Here, it often makes sense to use thread-pools which create a fixed upper number of connections and schedule work from a queue to run on these.

You could use eg Rails' ConnectionPool to handle the database connections and schedule your statements using one of the ThreadPool implementations from the excellent concurrent-ruby gem.