Encoding::UndefinedConversionError when calling Chef update command from a Rails application

Question

I'm building a Rails app that uses Chef-DK underthehood.

What I'm trying to achieve is to execute the same code as the chef update <path/to/policyfile> .

Here is the interesting part of the code :

require 'chef-dk/command/update'

chef_update = ChefDK::Command::Update.new
update_options = [policyfile_path]
update_options << '--debug' if Rails.env.development?
update = chef_update.run(update_options)

When this run, it start to update the policyfile but fails with the following output:

Attributes already up to date
Building policy k8sworker
Expanded run list: recipe[fail2ban], recipe[os-hardening], recipe[rkhunter], recipe[ssh-hardening], recipe[ssh-keys], recipe[tincvpn]
Caching Cookbooks...
Installing tincvpn ~> 0.1.10 from github
Installing fail2ban 6.0.0
Error: Failed to generate Policyfile.lock
Reason: (Encoding::UndefinedConversionError) "\x8B" from ASCII-8BIT to UTF-8
/usr/local/lib/ruby/2.6.0/delegate.rb:349:in `write'
/usr/local/lib/ruby/2.6.0/delegate.rb:349:in `block in delegating_block'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:528:in `block in stream_to_tempfile'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:497:in `call_block'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:488:in `<<'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:163:in `read'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:293:in `block in read_body_0'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:253:in `inflater'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:283:in `read_body_0'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:204:in `read_body'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:527:in `stream_to_tempfile'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:230:in `block in streaming_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http/basic_client.rb:92:in `block in request'
/usr/local/lib/ruby/2.6.0/net/http.rb:1518:in `block in transport_request'
/usr/local/lib/ruby/2.6.0/net/http/response.rb:165:in `reading_body'
/usr/local/lib/ruby/2.6.0/net/http.rb:1517:in `transport_request'
/usr/local/lib/ruby/2.6.0/net/http.rb:1479:in `request'
/usr/local/lib/ruby/2.6.0/net/http.rb:1472:in `block in request'
/usr/local/lib/ruby/2.6.0/net/http.rb:920:in `start'
/usr/local/lib/ruby/2.6.0/net/http.rb:1470:in `request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http/basic_client.rb:69:in `request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:370:in `block in send_http_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:411:in `block in retrying_http_errors'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:409:in `loop'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:409:in `retrying_http_errors'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:365:in `send_http_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:388:in `block (2 levels) in send_http_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:485:in `follow_redirect'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:383:in `block in send_http_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:411:in `block in retrying_http_errors'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:409:in `loop'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:409:in `retrying_http_errors'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:365:in `send_http_request'
/usr/local/bundle/gems/chef-15.1.36/lib/chef/http.rb:228:in `streaming_request'
/usr/local/bundle/gems/cookbook-omnifetch-0.8.1/lib/cookbook-omnifetch/artifactserver.rb:42:in `install'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile/cookbook_location_specification.rb:81:in `ensure_cached'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile_compiler.rb:181:in `block in install'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile_compiler.rb:176:in `each'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile_compiler.rb:176:in `install'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile_services/install.rb:101:in `generate_lock_and_install'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/policyfile_services/install.rb:63:in `run'
/usr/local/bundle/gems/chef-dk-4.0.60/lib/chef-dk/command/update.rb:93:in `run'
/application/app/interactors/node_actions/run_chef_update.rb:24:in `call'
...

And here is the Chef output with the trace log level:

worker_1     | [2019-07-03T08:02:17+00:00] TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: Initiating GET to https://supermarket.chef.io/universe
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: ---- HTTP Request Header Data: ----
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
worker_1     | [2019-07-03T08:02:17+00:00] TRACE: ---- End HTTP Request Header Data ----
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: ---- HTTP Status and Header Data: ----
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: HTTP 1.1 200 OK
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: cache-control: max-age=0, private, must-revalidate
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: content-encoding: gzip
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: content-type: application/json; charset=utf-8
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: date: Wed, 03 Jul 2019 08:02:19 GMT
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: etag: W/"461b5134d1467f6950f82af1510078a0"
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: server: nginx
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: x-content-type-options: nosniff
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: x-frame-options: SAMEORIGIN
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: x-request-id: 0bc2cc4a-0399-4953-93db-699176e11e36
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: x-runtime: 0.301233
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: x-xss-protection: 1; mode=block
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: transfer-encoding: chunked
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: connection: Close
worker_1     | [2019-07-03T08:02:20+00:00] TRACE: ---- End HTTP Status/Header Data ----
worker_1     | [2019-07-03T08:02:22+00:00] TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
worker_1     | [2019-07-03T08:02:22+00:00] TRACE: HTTP server did not include a Content-Length header in response, cannot identify truncated downloads.
worker_1     | [2019-07-03T08:02:22+00:00] TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
worker_1     | [2019-07-03T08:02:22+00:00] TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
worker_1     | [2019-07-03T08:02:22+00:00] TRACE: Decompressing gzip response
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: Initiating GET to https://supermarket.chef.io/api/v1/cookbooks/fail2ban/versions/6.0.0/download
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: ---- HTTP Request Header Data: ----
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
worker_1     | [2019-07-03T08:02:23+00:00] TRACE: ---- End HTTP Request Header Data ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- HTTP Status and Header Data: ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: HTTP 1.1 302 Found
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: cache-control: no-cache
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: content-type: text/html; charset=utf-8
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: date: Wed, 03 Jul 2019 08:02:25 GMT
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: location: https://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/28864/original/fail2ban20190508-41006-1kydj0z.tar.gz?1557352785
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: server: nginx
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-content-type-options: nosniff
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-frame-options: SAMEORIGIN
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-request-id: 17a0d07d-cf03-45b3-956f-8eb7fcabd2a7
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-runtime: 0.044161
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-xss-protection: 1; mode=block
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: content-length: 209
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: connection: Close
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- End HTTP Status/Header Data ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- HTTP Response Body ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: <html><body>You are being <a href="https://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/28864/original/fail2ban20190508-41006-1kydj0z.tar.gz?1557352785">redirected</a>.</body></html>
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- End HTTP Response Body -----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_stream_complete
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: No content-length information collected for the streamed download, cannot identify streamed download.
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP calling Chef::HTTP::CookieManager#handle_stream_complete
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP calling Chef::HTTP::Decompressor#handle_stream_complete
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Following redirect 1/10
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Initiating GET to https://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/28864/original/fail2ban20190508-41006-1kydj0z.tar.gz?1557352785
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- HTTP Request Header Data: ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- End HTTP Request Header Data ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- HTTP Status and Header Data: ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: HTTP 1.1 200 OK
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-amz-id-2: a2iOnve5iX8paa/pT2s+A21Bm8IvSqbjzYeP8mc2I2kNGSwgj9Wen8IyQlpiQ955grAF7xZKaQE=
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: x-amz-request-id: E74D7A5F59B4B3B0
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: date: Wed, 03 Jul 2019 08:02:26 GMT
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: last-modified: Wed, 08 May 2019 21:59:47 GMT
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: etag: "5c15105980cf9bfb2467ed00b789be0a"
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: accept-ranges: bytes
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: content-type: application/gzip
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: content-length: 9020
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: server: AmazonS3
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: connection: close
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: ---- End HTTP Status/Header Data ----
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Streaming download from https://supermarket.chef.io/api/v1/cookbooks/fail2ban/versions/6.0.0/download to tempfile /tmp/chef-rest20190703-1-cbdl7x
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: content_encoding = ''               initializing noop stream deflator.
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP::StreamHandler calling Chef::HTTP::ValidateContentLength::ContentLengthCounter#handle_chunk
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP::StreamHandler calling Chef::HTTP::Decompressor::NoopInflater#handle_chunk
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP::StreamHandler calling Chef::HTTP::ValidateContentLength::ContentLengthCounter#handle_chunk
worker_1     | [2019-07-03T08:02:25+00:00] TRACE: Chef::HTTP::StreamHandler calling Chef::HTTP::Decompressor::NoopInflater#handle_chunk

So it's like while deflating the fail2ban archive from Amazon S3, it fails due to some encoding stuff.

From that point:

1. when I then run the chef update command from the terminal, it works fine, no errors
2. when I run my code again, it works also without any errors (I guess due to the chef cached version of the cookbooks, so no more going through the failing code)

So to me it's like something is missing in my code in order to get this part working, but I can't figure out what could it be.

How could I solve this issue?

PS: The app is running in a Docker container with a Linux Alpine.

Answer 1

Chef library is using a Tempfile in order to write the downloaded data somewhere, so I aliased its write method in order to call force_encoding('UTF-8') on the passed argument before to give it back to the original write method.

Here is my override (in a config/initializers/tempfile_override.rb :

# frozen_string_literal: true

require 'tempfile'

#
# Overrides the Tempfile write method in order to always use `force_encode` with
# UTF-8 in order to solve the issue where `chef update` fails to write
# downloaded cookbooks data and write it to the disk.
# See https://stackoverflow.com/questions/56866746/encodingundefinedconversionerror-when-calling-chef-update-command-from-a-rails
#
class Tempfile
  alias old_write write
  def write(data)
    old_write(data.force_encoding('UTF-8'))
  end
end

I'm kind of forced to do that as I need to change the Chef::UI object in order to grab the Chef logs in my app.

That's not clean, but I don't have a better way to do it without having to patch chef itself.

Until something better has been found, this answer will be the accepted one.

If you know a better way to do it, please help me here :)