Controlling when participants are added with vanity a/b test

Question

Using vanity to conduct an A/B test. The test is reliant on a modal that is conditionally shown to certain users. The modal contains different incentives based on which side of the test you see.

The code for the modal is in a partial that gets loaded on several pages in our app.

Example:

<%= render partial: 'modals/example' %>

That partial contains a condition for the content that will be shown:

<% if ab_test(:whatever) == "something" %>
   <p>Stuff to show</p>
<% else %>
   <p>Different stuff to show</p>
<% end %>

I'm tracking the relevant metric in the relevant place.

The test itself works as expected in terms of showing the content and converting. The problem I'm running into is that because the partial is loaded even when the modal isn't show...everyone that lands on a page that references this partial is being added as a "participant" and I would like to only count people who have actually seen the modal as participants.

We use mixpanel as well and I can pivot into running the test with that. I like vanity, though, and would prefer to use it if there's a reasonable solution that isn't going to significantly increase the time it takes to implement. Was hoping someone SO could point out what I'm missing and how to approach solving.

Answer 1

I would try to send a post via ajax with the data needed to add a participant to /vanity/add_participant after the modal is shown.

$('#myModal').on('shown', function(){
    $.ajax({
    type: "POST",
    url: '/vanity/add_participant',
    data: {
      participant_data: participantData
    },
    success: function(data) {}
  })
});

Add the following to application.rb:

Vanity.configure do |config|
  config.use_js = true

  # Optionally configure the add_participant route that is added with Vanity::Rails::Dashboard,
  # make sure that this action does not require authentication
  # config.add_participant_route = '/vanity/add_participant'
end

Then add <%= vanity_js %> to any page that calls an A/B test after calling ab_test. vanity_js needs to be included after your call to ab_test so that it knows which version of the experiment the participant is a member of. The helper will render nothing if the there are no ab_tests running on the current page, so adding vanity_js to the bottom of your layouts is a good option. Keep in mind that if you set use_js and don't include vanity_js in your view no participants will be recorded.

Not sure if this helps haha.