problems with Scraping a Website with Elixir

Question

I'm trying to get a simple hound test working with my app, I figured out its an error with selenium. This is the code:

In mix.exs:

 defmodule Scraper.Mixfile do use Mix.Project def project do [app: :scraper, version: "0.0.1", elixir: "~> 1.0", build_embedded: Mix.env == :prod, start_permanent: Mix.env == :prod, deps: deps] end # Configuration for the OTP application # # Type `mix help compile.app` for more information def application do [applications: [:logger, :httpoison, :hound]] end # Dependencies can be Hex packages: # # {:mydep, "~> 0.3.0"} # # Or git/path repositories: # # {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"} # # Type `mix help deps` for more examples and options defp deps do [ {:httpoison, "~> 0.7"}, {:floki, "~> 0.7"}, {:hound, "~> 0.7"} ] end end 

In lib/scraper.ex

 defmodule Example do use Hound.Helpers def run do Hound.start_session IO.inspect "Iniciando" navigate_to "http://akash.im" IO.inspect page_title() Hound.end_session end end 

In config/config.exs

 # This file is responsible for configuring your application # and its dependencies with the aid of the Mix.Config module. use Mix.Config # This configuration is loaded before any dependency and is restricted # to this project. If another project depends on this project, this # file won't be loaded nor affect the parent project. For this reason, # if you want to provide default values for your application for third- # party users, it should be done in your mix.exs file. # Sample configuration: # # config :logger, :console, # level: :info, # format: "$date $time [$level] $metadata$message\\n", # metadata: [:user_id] # It is also possible to import configuration files, relative to this # directory. For example, you can emulate configuration per environment # by uncommenting the line below and defining dev.exs, test.exs and such. # Configuration from the imported file will override the ones defined # here (which is why it is important to import them last). # # import_config "#{Mix.env}.exs" # Define how long the application will wait between failed attempts (in miliseconds) config :hound, retry_time: 100000 # Start with selenium driver (default) config :hound, driver: "selenium" 

Starting a webdriver server

 java -jar selenium-server-standalone-2.45.0.jar 

Run app:

 /scraper$ iex -S mix Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false] Interactive Elixir (1.0.5) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> Example.run ** (exit) exited in: :gen_server.call(Hound.SessionServer, {:find_or_create_session, #PID<0.148.0>}, 60000) ** (EXIT) an exception was raised: ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}} (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4 (hound) lib/hound/session_server.ex:22: Hound.SessionServer.handle_call/3 (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4 (stdlib) gen_server.erl:661: :gen_server.handle_msg/5 (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3 11:26:13.971 [error] GenServer Hound.SessionServer terminating Last message: {:find_or_create_session, #PID<0.148.0>} State: #HashDict<[]> ** (exit) an exception was raised: ** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}} (hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4 (hound) lib/hound/session_server.ex:22: Hound.SessionServer.handle_call/3 (stdlib) gen_server.erl:629: :gen_server.try_handle_call/4 (stdlib) gen_server.erl:661: :gen_server.handle_msg/5 (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3 (stdlib) gen_server.erl:212: :gen_server.call/3 (scraper) lib/scraper.ex:37: Example.run/0 iex(1)> 

Answer 1

The request timed out in this case, as can be seen from the line

** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: :timeout}}

If you look at the stack trace, it indicates the error is at

(hound) lib/hound/request_utils.ex:43: Hound.RequestUtils.send_req/4

And if you open up hound source, on line 43 of lib/hound/request_utils.ex you see

case type do
  :get ->
    {:ok, resp} = HTTPoison.get(url, headers, @http_options)
  :post ->
    {:ok, resp} = HTTPoison.post(url, body, headers, @http_options)
  :delete ->
    {:ok, resp} = HTTPoison.delete(url, headers, @http_options)
end

This code expects a response, and crashes otherwise. There's a timeout error in your case, causing the crash.

Please check if the website up and reachable when you run the test, and retry.