What's the most efficient way to keep a user database in sync with an external mailing list service?

Question

I'd like some advice on how I should synchronize a list of email addresses on 11k users against an external mailing list program, in this case Mailchimp.

Normally the way I'd do this is simply to have an :after_save callback, to send a single update to the external api.

But already each hour, a rake task is run to update a property on every user in the database. If I simply did that, every hour, the the poor mailchimp API would get be hit 11,000 times.

What's the most efficient, simple way to do this, to check only if a single attribute you're watching has changed from what it was before the save?

If there's a variable that persists across the transaction lifecycle I would simply do something like this, where I check if the value has changed, and if it's different execute come other code.

class User

  :before_save :store_old_email

  :after_save :sync_with_chimp


  def store_old_email
    $ugly_of_global_variable_to_store_email = user.email
  end

  :sync_with_chimp 
    if $ugly_of_global_variable_to_store_email != user.email
      //update_mail_chimp_api
    end
  end

end

I've checked the rails api here, and I'm still slightly unclear on how I should be doing this.

Would you use the dirty? class here to do this?

Answer 1

This is the way I went with in the end.

It turns out Rails gives you loads of handy callbacks in the dirty to do this.

Any suggestions on how to make this code less repetitive wold be gratefully received.

def update_mailchimp(optin)
  # Create a Hominid object (A wrapper to the mailchimp api), and pass in a hash from the yaml file 
  # telling which mailing list id to update with subscribe/unsubscribe notifications)
  @hominid = Hominid.new
  client_site_list_id = YAML.load(File.read(RAILS_ROOT + "/config/mailchimp.yml"))

  case optin  
    when 'subscribe_newsletter'
      logger.debug("subscribing to newsletter...")
      "success!" if @hominid.subscribe(client_site_list_id['client_site_to_mailchimp_API_link'], email, {:FNAME => first_name, :LNAME => last_name}, 'html')
    when 'unsubscribe_newsletter'
      logger.debug("unsubscribing from newsletter...")
      "success!" if @hominid.subscribe(client_site_list_id['client_site_to_mailchimp_API_link'], email, {:FNAME => first_name, :LNAME => last_name}, 'html')
    when 'subscribe_monthly_update'
      logger.debug("subscribing to monthly update...")
      "success!" if @hominid.subscribe(client_site_list_id['monthly_update'], email, {:FNAME => first_name, :LNAME => last_name}, 'html')
    when 'unsubscribe_monthly_update'
      logger.debug("unsubscribing from monthly update...")
      "success!" if @hominid.unsubscribe(client_site_list_id['monthly_update'], email, {:FNAME => first_name, :LNAME => last_name}, 'html')
    end
end

# Keep the users in sync with mailchimp's own records - by only firing requests to the API if details on a user have changed after saving.

def check_against_mailchimp
  logger.info("Checking if changes need to be sent to mailchimp...")
  if newsletter_changed?
    logger.info("Newsletter changed...")
    newsletter ? update_mailchimp('subscribe_newsletter') : update_mailchimp('unsubscribe_newsletter')
  end
  if monthly_update_changed?
    logger.info("update preferences changed...")
    monthly_update ? update_mailchimp('subscribe_monthly_update') : update_mailchimp('unsubscribe_monthly_update')
  end
end

Answer 2

you could change your users model to an active resource instead of active record and just use mailchimps api as your db for users

this is an older post about active resource but might get you started down the right path

http://www.therailsway.com/2007/9/3/using-activeresource-to-consume-web-services