I am trying to implement a multithreaded method in rails such that I can create/update multiple records very quickly.
This is the outline of my program.
ActiveRecord::Base.transaction do
(1..10).each do |i|
arr[i] = Thread.new {
<some logic on obj>
...
...
obj.save!
}
end
arr.each {|t| t.join}
end
This gives me warnings on my log.
DEPRECATION WARNING: Database connections will not be closed automatically,
please close your database connection at the end of the thread by calling `close`
on your connection.
And it gives me an error
could not obtain a database connection within 5 seconds (waited 5.059358 seconds).
The max pool size is currently 5; consider increasing it.
I tried: - changing database.yaml and increasing the poolsize and timeout there. - modified the existing code in the following way.
ActiveRecord::Base.connection_pool.clear_stale_cached_connections!
begin
ActiveRecord::Base.transaction do
(1..10).each do |i|
arr[i] = Thread.new {
<some logic on obj>
...
...
obj.save!
ActiveRecord::Base.connection.close
}
end
arr.each {|t| t.join}
end
ensure
ActiveRecord::Base.connection.close if ActiveRecord::Base.connection
ActiveRecord::Base.clear_active_connections!
end
I am still getting the same warning and error. I am obviously missing the concept here. How do I proceed with this?
To prevent connection leak in multi threads, you have to manage the connection manually. You can try:
Thread.new do
ActiveRecord::Base.connection_pool.with_connection do
# Do whatever
end
end
One problem of ActiveRecord::Base.connection_pool.with_connection
is that it always prepare a connection even if the code inside does not need it.
We can improve it a little bit by using ActiveRecord::Base.connection_pool.release_connection
:
Thread.new do
begin
# Do whatever
ensure
ActiveRecord::Base.connection_pool.release_connection
end
end
if you want to run in N threads, make sure you have N connections in the pool
# ActiveRecord::Base.connection_config
# # => {pool: 5, timeout: 5000, database: "db/development.sqlite3", adapter: "sqlite3"}
otherwise you will be able to fetch a connection from the pool .
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.