Using Ruby Mongo with ActiveJob

Question

I am using Ruby 2.7 with Mongo 2.17 client. Currently using Sidekiq with ActiveJob to perform millions of Jobs executions to do single transactions to AWS DocumentDB. While reading the Mongo client documentation I see that they claim that is a bad idea to instantiate a Client per request, rather than just having 1 and reusing it.

Currently the job that runs millions of times does instantiate a client and closes it at the end, the job has many threads executing per Sidekiq process, currently running multiple Sidekiq processes:

jobs/my_job.rb
def perform(document)
   client = Mongo::Client.new(DOCUMENTDB_HOST, DOCUMENTDB_OPTIONS)
   client.insert_one(document)
   client.close
end

From the documentation it states :

The default configuration for a Mongo::Client works for most applications:

client = Mongo::Client.new(["localhost:27017"])

Create this client once for each process, and reuse it for all operations. It is a common mistake to create a new client for each request, which is very inefficient and not what the client was designed for.
To support extremely high numbers of concurrent MongoDB operations within one process, increase max_pool_size:

client = Mongo::Client.new(["localhost:27017"], max_pool_size: 200)

Any number of threads are allowed to wait for connections to become available, and they can wait the default (1 second) or the wait_queue_timeout setting:

client = Mongo::Client.new(["localhost:27017"], wait_queue_timeout: 0.5)

When #close is called on a client by any thread, all connections are closed:

client.close

Note that when creating a client using the block syntax described above, the client is automatically closed after the block finishes executing.

My question would be if this statement also applies for isolated Sidekiq jobs execution, and if so, how could i recycle Mongo Client connection object along a Sidekiq process? I could think of having a global @@client in the Sidekiq initializer:

config/initializers/sidekiq.rb:

@@client = Mongo::Client.new(DOCUMENTDB_HOST, DOCUMENTDB_OPTIONS)

and then:

jobs/my_job.rb:

def perform(document)
   @@client[:my_collection].insert_one(document)
end

Note: No significant errors are raised, the whole system just get frozen and I get the following exception thrown randomly after the system has several minutes running correctly:

OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello (for 10.0.0.123:27017

UPDATE:

I tried 'reusing' the connection client by creating a global variable with the connection object in an initializer:

config/initializers/mongodb_client.rb
$mongo_client = Mongo::Client.new(DOCUMENTDB_HOST, DOCUMENTDB_OPTIONS)

and then using it inside my ActiveJob class. So far it seems to work good but I am unaware of side effects; actually I did start many Sidekiq processes and I am closely watching at the logs looking for exceptions thrown, so far all good.

jobs/my_job.rb
def perfom(document)
   $mongo_client[:activity_log].insert_one(log_document)
end

Answer 1

It looks like the MongoClient is threadsafe, just set that:max_pool_size to your Sidekiq concurrency so each job thread can concurrently use the client.