Searchkick with aws elasticsearch

Question

Im having a problem with searchkick and elasticsearch and using the faraday_middleware-aws-signers-v4 gem. If I leave my elasticsearch.rb initializer blank, it works on my development side. However if I put in the configuration for AWS Elasticsearch I get a json parse error. Here is my elasticsearch.rb file:

require 'faraday_middleware/aws_signers_v4'

Searchkick.aws_credentials = {
  access_key_id: ENV["AWS_ACCESS_KEY_ID"],
  secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
  region: "us-west-2"
}

I followed the instructions on the github page for AWS. I left out the ENV['ELASTICSEARCH_URL'] because that is defined in my env files. I tried it with it in there and it gave the same results. Ive also tried it this way, which was the old way it was suppose to have it and get same results:

require 'faraday_middleware/aws_signers_v4'

Searchkick.client =
  Elasticsearch::Client.new(
    url: ENV['ELASTICSEARCH_URL'],
    transport_options: {request: {timeout: 10}}
  ) do |f|
    f.request :aws_signers_v4, {
      credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
      service_name: "es",
      region: "us-west-2"
    }
  end

Ive been able to narrow it down to this configuration though.

Edit: Heres the full error output:

livingrecipe_1   | Oj::ParseError - unexpected character at line 1, column 1 [parse.c:690]:
livingrecipe_1   |   multi_json (1.12.2) lib/multi_json/adapters/oj.rb:15:in `load'
livingrecipe_1   |   multi_json (1.12.2) lib/multi_json/adapter.rb:21:in `load'
livingrecipe_1   |   multi_json (1.12.2) lib/multi_json.rb:122:in `load'
livingrecipe_1   |   elasticsearch-transport (5.0.4) lib/elasticsearch/transport/transport/serializer/multi_json.rb:24:in `load'
livingrecipe_1   |   elasticsearch-transport (5.0.4) lib/elasticsearch/transport/transport/base.rb:322:in `perform_request'
livingrecipe_1   |   elasticsearch-transport (5.0.4) lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
livingrecipe_1   |   elasticsearch-transport (5.0.4) lib/elasticsearch/transport/client.rb:131:in `perform_request'
livingrecipe_1   |   elasticsearch-api (5.0.4) lib/elasticsearch/api/actions/info.rb:15:in `info'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick.rb:77:in `server_version'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick.rb:81:in `server_below?'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/query.rb:906:in `below50?'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/query.rb:723:in `set_order'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/query.rb:437:in `prepare'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/query.rb:43:in `initialize'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick.rb:95:in `search'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/model.rb:29:in `searchkick_search'
livingrecipe_1   |   app/helpers/recipe_search.rb:18:in `search'
livingrecipe_1   |   app/controllers/recipes_controller.rb:9:in `index'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
livingrecipe_1   |   actionpack (5.1.3) lib/abstract_controller/base.rb:186:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/rendering.rb:30:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/callbacks.rb:131:in `run_callbacks'
livingrecipe_1   |   actionpack (5.1.3) lib/abstract_controller/callbacks.rb:19:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/rescue.rb:20:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/notifications.rb:166:in `block in instrument'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/notifications.rb:166:in `instrument'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
livingrecipe_1   |   searchkick (2.3.1) lib/searchkick/logging.rb:209:in `process_action'
livingrecipe_1   |   activerecord (5.1.3) lib/active_record/railties/controller_runtime.rb:22:in `process_action'
livingrecipe_1   |   actionpack (5.1.3) lib/abstract_controller/base.rb:124:in `process'
livingrecipe_1   |   actionview (5.1.3) lib/action_view/rendering.rb:30:in `process'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal.rb:189:in `dispatch'
livingrecipe_1   |   actionpack (5.1.3) lib/action_controller/metal.rb:253:in `dispatch'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/routing/route_set.rb:31:in `serve'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/routing/mapper.rb:16:in `block in <class:Constraints>'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/routing/mapper.rb:46:in `serve'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/journey/router.rb:46:in `block in serve'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/journey/router.rb:33:in `serve'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/routing/route_set.rb:834:in `call'
livingrecipe_1   |   rack-pjax (1.0.0) lib/rack/pjax.rb:12:in `call'
livingrecipe_1   |   omniauth (1.6.1) lib/omniauth/strategy.rb:189:in `call!'
livingrecipe_1   |   omniauth (1.6.1) lib/omniauth/strategy.rb:167:in `call'
livingrecipe_1   |   omniauth (1.6.1) lib/omniauth/strategy.rb:189:in `call!'
livingrecipe_1   |   omniauth (1.6.1) lib/omniauth/strategy.rb:167:in `call'
livingrecipe_1   |   omniauth (1.6.1) lib/omniauth/builder.rb:63:in `call'
livingrecipe_1   |   remotipart (1.3.1) lib/remotipart/middleware.rb:32:in `call'
livingrecipe_1   |   clearance (1.16.0) lib/clearance/rack_session.rb:23:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/etag.rb:25:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/conditional_get.rb:25:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/head.rb:12:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/session/abstract/id.rb:232:in `context'
livingrecipe_1   |   rack (2.0.3) lib/rack/session/abstract/id.rb:226:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/cookies.rb:613:in `call'
livingrecipe_1   |   activerecord (5.1.3) lib/active_record/migration.rb:556:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/callbacks.rb:97:in `run_callbacks'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/callbacks.rb:24:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/executor.rb:12:in `call'
livingrecipe_1   |   better_errors (2.3.0) lib/better_errors/middleware.rb:84:in `protected_app_call'
livingrecipe_1   |   better_errors (2.3.0) lib/better_errors/middleware.rb:79:in `better_errors_call'
livingrecipe_1   |   better_errors (2.3.0) lib/better_errors/middleware.rb:57:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
livingrecipe_1   |   web-console (3.5.1) lib/web_console/middleware.rb:135:in `call_app'
livingrecipe_1   |   web-console (3.5.1) lib/web_console/middleware.rb:20:in `block in call'
livingrecipe_1   |   web-console (3.5.1) lib/web_console/middleware.rb:18:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
livingrecipe_1   |   railties (5.1.3) lib/rails/rack/logger.rb:36:in `call_app'
livingrecipe_1   |   railties (5.1.3) lib/rails/rack/logger.rb:24:in `block in call'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/tagged_logging.rb:69:in `block in tagged'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/tagged_logging.rb:26:in `tagged'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/tagged_logging.rb:69:in `tagged'
livingrecipe_1   |   railties (5.1.3) lib/rails/rack/logger.rb:24:in `call'
livingrecipe_1   |   sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:13:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
livingrecipe_1   |   request_store (1.3.2) lib/request_store/middleware.rb:9:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/request_id.rb:25:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/method_override.rb:22:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/runtime.rb:22:in `call'
livingrecipe_1   |   rack-timeout (0.4.2) lib/rack/timeout/core.rb:122:in `block in call'
livingrecipe_1   |   rack-timeout (0.4.2) lib/rack/timeout/support/timeout.rb:19:in `timeout'
livingrecipe_1   |   rack-timeout (0.4.2) lib/rack/timeout/core.rb:121:in `call'
livingrecipe_1   |   activesupport (5.1.3) lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/executor.rb:12:in `call'
livingrecipe_1   |   actionpack (5.1.3) lib/action_dispatch/middleware/static.rb:125:in `call'
livingrecipe_1   |   rack (2.0.3) lib/rack/sendfile.rb:111:in `call'
livingrecipe_1   |   railties (5.1.3) lib/rails/engine.rb:522:in `call'
livingrecipe_1   |   puma (3.10.0) lib/puma/configuration.rb:225:in `call'
livingrecipe_1   |   puma (3.10.0) lib/puma/server.rb:605:in `handle_request'
livingrecipe_1   |   puma (3.10.0) lib/puma/server.rb:437:in `process_client'
livingrecipe_1   |   puma (3.10.0) lib/puma/server.rb:301:in `block in run'
livingrecipe_1   |   puma (3.10.0) lib/puma/thread_pool.rb:120:in `block in spawn_thread'

Answer 1

For anyone who might stumble upon this and need the answer. This had to do with the way Elasticsearch 5 returns the request in a gzip compressed format. By using the faraday_middleware gem and setting it in the initializer like so:

require 'faraday/middleware'
require 'faraday_middleware/aws_signers_v4'

Searchkick.client =
  Elasticsearch::Client.new(
    url: ENV["ELASTICSEARCH_URL"],
    transport_options: {request: {timeout: 10}}
  ) do |f|
    f.use FaradayMiddleware::Gzip
    f.request :aws_signers_v4, {
      credentials: Aws::Credentials.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_ACCESS_KEY"]),
      service_name: "es",
      region: "us-west-2"
    }
  end

It was able to handle the return request perfectly. See comments above for reference to post on the searchkick github page