同步不同Sidekiq線程的方法並等待

Question

問題：我有多個sidekiq線程和一個函數，一次只能從任何線程中調用一次。

原因：我們正在查詢AdWords API以獲取一些數據。 當涉及到速率限制時，它們是非常嚴格的。 線程中只有一個可以調用該函數一次獲取數據。

現在一些代碼：

# Public: Get estimates for a set of keywords. If there is an error, retry
# several times. If not successful, raise an error
#
# keywords: The keyword objects to get estimates for.
# save: Boolean to indicate whether the keyword objects should be saved to
# the database
#
def repeatedly_try_get_estimates(keywords: [], save: true, sleep_delay: 150)
  return keywords if keywords.empty?
  func = -> { get_estimates(keywords, !save) }
  retry_operation(function: func, max_tries: 15, sleep_delay: sleep_delay)
end

• 如您所見，現在我在解決該問題方面有大量的sleep_delay工作。
• 該代碼以get_estimates函數為參數調用retry_operation函數。 然后get_estimates多次重試get_estimates函數，直到出現API異常。

retry_function ：

# Private: Retry a function X times and wait X seconds. If it does not work X times,
# raise an error. If successful return the functions results.
#
# - max_tries: The maximum tries to repeat the function
# - sleep_delay: The seconds to wait between each iteration.
# - function: The lambda function to call each iteration
#
def retry_operation(max_tries: 5, sleep_delay: 30, function: nil, current_try: 0, result: nil)

  # Can't call, no function
  if function.nil?
    return
  end

  # Abort, tried too frequently.
  if current_try > max_tries
    raise "Failed function too often"
  end

  # Check if there is an exception
  exception = true
  begin
    result = function.call
    exception = false
  rescue => e
    Rails.logger.info "Received error when repeatedly calling function #{e.message.to_s}"
  end

  if exception
    sleep sleep_delay if sleep_delay > 0
    retry_operation(max_tries: max_tries, sleep_delay: sleep_delay, function: function, current_try: current_try + 1)
  else
    result
  end
end

get_estimates_function在這里： https : get_estimates_function 。 太長了，以防萬一。

我想我需要執行以下操作：

1. 調整repeatedly_try_get_estimates函數中的代碼。
2. 在類中使用互斥鎖。
3. 如果正在使用互斥鎖，請拯救異常。
4. 僅當互斥鎖可用時，才運行rety_operation ，否則請休眠一段時間

謝謝你的幫助 ：）

Answer 1

現在我們開始工作了：

# Public: Get estimates for a set of keywords. If there is an error, retry
# several times. If not successful, raise an error
#
# keywords: The keyword objects to get estimates for.
# save: Boolean to indicate whether the keyword objects should be saved to
# the database
#
def repeatedly_try_get_estimates(keywords: [], save: true, sleep_delay: 40)
  return keywords if keywords.empty?
  func = -> { get_estimates(keywords, save_keywords: true) }
  exception = nil
  result = nil
  initial_sleep = 0

  estimates_mutex.synchronize do
    since_last_request = Time.now.to_i - last_adwords_api_request
    if since_last_request <= 30
      Rails.logger.info "AdWords: Last request was only few seconds ago - sleeping #{since_last_request}."
      initial_sleep = since_last_request
    end
    begin
      result = retry_operation(function: func, max_tries: 15, sleep_delay: sleep_delay, initial_sleep: initial_sleep)
    rescue => e
      exception = e
    end
    @@last_adwords_api_request = Time.now.to_i
  end
  if exception
    raise exception
  end
  result
end