How do I use my own cookies in capybara?

Question

I'm trying to (ab)use the capybara web testing framework to automate some tasks on github that are not accessible via the github API and which require me to be logged in and click on buttons to send AJAX requests.

Since capybara/selenium is a testing framework it helpfully creates a temporary session which has no cookies in it. I'd like to either stop it from doing that, or else I'd like to know how to load my cookie store into the browser session that it creates.

All I'm trying to do is this:

#!/usr/bin/env ruby

require 'selenium-webdriver'

driver = Selenium::WebDriver.for :chrome
driver.navigate.to "https://github.com"

Or this:

#!/usr/bin/env ruby

require 'capybara'

Capybara.register_driver :selenium do |app|
  Capybara::Selenium::Driver.new(app, :browser => :chrome)
end

session = Capybara::Session.new(:selenium)
session.visit "https://www.github.com"

In both cases I get the github.com landing page you'd see as a logged-out user or incognito mode in the browser. I'd like to get my logged-in landing page like I just fired up a web browser myself and navigated to that URL.

Since I have 2FA setup on github that makes automating the login process from the github landing page somewhat annoying, so I'd like to avoid automating logging into github. The tasks that I want to automate do not require re-authenticating via 2FA.

ANSWER:

For MacOSX+Ruby+Selenium this works:

#!/usr/bin/env ruby

require 'selenium-webdriver'

caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"}, detach: false)
driver = Selenium::WebDriver.for :chrome, :desired_capabilities => caps
driver.navigate.to "https://github.com"

Then fire up chrome with this:

% /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/Users/lamont/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480

Obviously the paths will need to be adjusted because they're OSX-centric and have my homedir in them.

There is also a bug in the selenium-webdriver gem for ruby where it inserts a 'detach' option which gets into a fight with 'debuggerAddress':

/Users/lamont/.rvm/gems/ruby-2.2.4/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/remote/response.rb:70:in `assert_ok': unknown error: cannot parse capability: chromeOptions (Selenium::WebDriver::Error::UnknownError)
from unknown error: unrecognized chrome option: detach

The lib/selenium/webdriver/chrome/bridge.rb file can be edited to take that out as a quick hack:

      chrome_options['binary']                   = Chrome.path if Chrome.path
      chrome_options['nativeEvents']             = true if native_events
      chrome_options['verbose']                  = true if verbose
      #chrome_options['detach']                   = detach.nil? || !!detach
      chrome_options['noWebsiteTestingDefaults'] = true if no_website_testing_defaults
      chrome_options['prefs']                    = prefs if prefs

Answer 1

To implement something similar in Ruby, check out this page that goes over that. Thanks to lamont for letting me know in the comments.

You can start chrome using a specific Chrome profile. I am not sure what the ruby implementation would look like, but in python it looks something like:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

options = ChromeOptions()
# more on this line here later.
options.add_experimental_option('debuggerAddress', '127.0.0.1:7878')
driver = webdriver.Chrome(chrome_options=otpions)

In order for this to work you need to do a few things.

manually start chrome from terminal/command prompt with these command line arguments --user-data-dir=/path/to/any/custom/directory/home/user/Desktop/Chromedir --profile-directory="Profile 1" --remote-debugging-port=7878

1. make sure "Profile 1" is already existing in the same --user-data-dir (make sure user Profile 1 has necessary chrome://components/ to run any apps that require those components)
2. you can use any free port in place of 7878 verify that http://localhost:7878 is running and returns value.

This should manually launch chrome with the "Profile 1" profile, and so long as it has logged into the site in question, it will stay logged in like a normal user so long as you follow these instructions to run the tests.

I used this to write a quick netflix bot that clicks the "continue playing" button when it pops up, and it's the only way to get DRM content to play as far as I have found. But it retains the cookies for the login, and also launches chrome with whatever components the profile is set up to have.

I have tried launching chrome with specific profiles before using different methodologies, but this was the only way to really force it to work how I wanted it to.

Edit: There are methods for saving cookie info as well although I don't know how well they work. Check out this link for more info, as my solution is probably not the best solution even if it works.

Answer 2

The show_me_the_cookies gem provides cross-driver cookie manipulation and can let you add new cookies. The one thing to be aware of when using selenium is that you need to visit the domain before you can create cookie for it, so you'll need to do something like

visit "https://www.github.com"
create_cookie(...)
visit "https://www.github.com"

for it to work - first visit just puts the browser/driver in a state where you can create the cookie, second visit actually goes to the page with the cookies set.

Answer 3

I had to tweak the OP's answer (from within her question) to get this going with Ruby in 2022.

Prerequisites

Chromedriver installed and allowed to run even though it's not signed:

> brew install chromedriver
> xattr -d com.apple.quarantine /usr/local/bin/chromedriver

Chrome launched and accepting commands on a specific port:

> /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=~/Library/Application\ Support/Google/Chrome --profile-directory=Default --remote-debugging-port=20480

This created a new profile in Chrome so I signed in to my account and got the browser set up, ready to start interacting with the (legacy EdTech) site I'm trying to automate.

Actual use

require 'selenium-webdriver'

caps = Selenium::WebDriver::Remote::Capabilities.chrome("goog:chromeOptions" => {"debuggerAddress" => "127.0.0.1:20480"})
driver = Selenium::WebDriver.for :chrome, capabilities: caps

driver.navigate.to "https://www.google.com"