In the second run of chef-client, lazy attribute of docker_container on “link” is not getting resolved and passed unreadable value

Question

my first time to ask stuff on stack-overflow, here in china i seldom able to meet any chef developer to talk about my problem so i am posting it here to seek for help. This issue has been bothering me for weeks and i am still trying to settle it.

here are my error msg:

  * directory[/root/tools/projectname/../bootproxy] action create (up to date)

* template[/root/tools/projectname/../bootproxy/oc.proxy.conf] action create (up to date) * directory[/root/tools/projectname/../bootproxy] action create (up to date) * file[/tmp/dockerinfo.txt] action delete - delete file /tmp/dockerinfo.txt * bash[docker ps -a|grep -v CONTAINER|grep -v monitor|awk '{print $1, $NF}'] action run - execute "bash" "/tmp/chef-script20170319-2107-fx41as" * ruby_block[result] action run - execute the ruby block result * docker_container[bootproxy] action redeploy - stopping bootproxy (will kill after 30s) - deleting bootproxy ================================================================================ Error executing action redeploy on resource 'docker_container[bootproxy]' ================================================================================

Docker::Error::ServerError
--------------------------
Could not get container for bootproxy

Cookbook Trace:
---------------
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:319:in `block (3 levels) in <class:DockerContainer>'
/var/chef/cache/cookbooks/docker/libraries/helpers_base.rb:66:in `with_retries'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:250:in `block (2 levels) in <class:DockerContainer>'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:81:in `converge_if_changed'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:247:in `block in <class:DockerContainer>'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `instance_eval'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `compile_and_converge_action'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:169:in `call_action'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:360:in `block in <class:DockerContainer>'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `instance_eval'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `compile_and_converge_action'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:169:in `call_action'
/var/chef/cache/cookbooks/docker/libraries/docker_container.rb:403:in `block in <class:DockerContainer>'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `instance_eval'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/copied_from_chef/chef/provider.rb:132:in `compile_and_converge_action'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:78:in `run_action'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block (2 levels) in converge'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `each'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block in converge'
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:105:in `converge'

Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/webserver/recipes/default.rb

218:         docker_container container_name do
219:           repo docker[:image]
220:           tag docker[:tag]
221:           #Add all docker link
222:           # links node.set[:linking]
223:           links lazy{node.run_state[:linking]}
224:           env docker[:env]
225:           command docker[:command]
226:           kill_after 30
227:     #      autoremove true
228:           action :redeploy
229:           port docker[:ports]
230:           volumes node.default["bindvolume"]
231:           cap_add 'SYS_ADMIN'
232:           devices []
233:           privileged true
234:           timeout 30
235:     #      {["/dev/fuse"]}
236:         end
237:       else
238:         docker_container container_name do
239:           repo docker[:image]
240:           tag docker[:tag]
241:           #Add all docker link
242:           links node.run_state[:linking]
243:           env docker[:env]
244:           command docker[:command]
245:           kill_after 30
246:     #      autoremove true
247:           action :redeploy
248:           port docker[:ports]
249:           volumes node.default["bindvolume"]
250:           cap_add 'SYS_ADMIN'
251:           devices []
252:           privileged true
253:           timeout 30
254:     #      {["/dev/fuse"]}
255:         end
256:       end
257: 
258:       if (not (defined?(docker[:exec])).nil?) && (not "#{docker[:exec]}" == "")
259:         execute 'execute command inside docker' do
260:         command "docker exec -i #{container_name} /bin/bash -c \'#{docker[:exec]}\'"
261:         end
262:       end
263: 
264:       etchosts.push("#{container_name}:#{container_name}")
265:     end
266: 
267:     #Add proxy.conf to folder if bootproxy defined
268:     if defined?(node[:externalmode]) && node[:externalmode].eql?("bootproxy")
269:       #Prepare bootproxy directories
270:       directory "#{node[:deploycode][:basedirectory]}../bootproxy" do

Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/webserver/recipes/default.rb:218:in `block in from_file'

docker_container("bootproxy") do
  action [:redeploy]
  retries 0
  retry_delay 2
  default_guard_interpreter :default
  declared_type :docker_container
  cookbook_name "webserver"
  recipe_name "default"
  kill_after 30
  repo "daocloud.io/library/nginx"
  tag "stable-alpine"
  exposed_ports {"80/tcp"=>{}, "5000/tcp"=>{}}
  port_bindings {"80/tcp"=>[{"HostIp"=>"0.0.0.0", "HostPort"=>"80"}], "5000/tcp"=>[{"HostIp"=>"0.0.0.0", "HostPort"=>"5000"}]}
  port ["80:80", "5000:5000"]
  volumes_binds ["/root/tools/projectname/../bootproxy:/etc/nginx/conf.d/"]
  links #<ChefCompat::CopiedFromChef::Chef::DelayedEvaluator:0x000000055c4a90@/var/chef/cache/cookbooks/webserver/recipes/default.rb:223>
  cap_add ["SYS_ADMIN"]
  privileged true
  timeout 30
  connection #<Docker::Connection:0x00000008301238 @url="unix:///", @options={:socket=>"/var/run/docker.sock", :read_timeout=>60}>
  network_mode "bridge"
  detach true
  signal "SIGTERM"
end

Running handlers: [2017-03-19T21:20:15+08:00] ERROR: Running exception handlers Running handlers complete [2017-03-19T21:20:15+08:00] ERROR: Exception handlers complete Chef Client failed. 20 resources updated in 34 seconds [2017-03-19T21:20:15+08:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [2017-03-19T21:20:15+08:00] ERROR: docker_container[bootproxy] (webserver::default line 218) had an error: Docker::Error::ServerError: Could not get container for bootproxy

[2017-03-19T21:20:15+08:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

and here are my codes:

  node.run_state[:linking] = []
  #Special handling if bootproxy,  get all local running docker id and name and link into bootproxy
  if localfolder.eql?("bootproxy")
    container_name = localfolder
    node.set[:dockerinfo] = []
    results = "/tmp/dockerinfo.txt"
    file results do
      action :delete
    end

    cmd = "docker ps -a|grep -v CONTAINER|grep -v monitor|awk \'{print $1, $NF}\'"
    bash cmd do
      code <<-EOH
      #{cmd} > #{results}
      EOH
    end

    ruby_block "result" do
      only_if { "cat #{results}| wc -l;while [ $? -ne 0 ]; do cat #{results}| wc -l;done" }
      # only_if { ::File.exists?(results) }
      block do
        f = File.open(results)
        dockerinfo = Hash.new
        f.each do |line|
          dockerinfo[line.chomp.split(' ')[0]] = line.chomp.split(' ')[1]
        end
        f.close
        node.set[:dockerinfo] = dockerinfo
        node.run_state[:linking] = dockerinfo
        node.run_state[:linking] = []
        node.set[:dockerinfo].each do |hash, dockername|
          node.run_state[:linking].push("#{dockername}:#{dockername}")
        end
      end
    end
  else
    node.run_state[:linking] = etchosts
  end

  if node.default["bindvolume"].eql?([":"])
    node.default["bindvolume"] = nil
  end
  if localfolder.eql?("bootproxy")
    # Using lazy evaluation if bootproxy
    docker_container container_name do
      repo docker[:image]
      tag docker[:tag]
      #Add all docker link
      # links node.set[:linking]
      links lazy{node.run_state[:linking]}
      env docker[:env]
      command docker[:command]
      kill_after 30
#      autoremove true
      action :redeploy
      port docker[:ports]
      volumes node.default["bindvolume"]
      cap_add 'SYS_ADMIN'
      devices []
      privileged true
      timeout 30
#      {["/dev/fuse"]}
    end
  else
    docker_container container_name do
      repo docker[:image]
      tag docker[:tag]
      #Add all docker link
      links node.run_state[:linking]
      env docker[:env]
      command docker[:command]
      kill_after 30
#      autoremove true
      action :redeploy
      port docker[:ports]
      volumes node.default["bindvolume"]
      cap_add 'SYS_ADMIN'
      devices []
      privileged true
      timeout 30
#      {["/dev/fuse"]}
    end
  end

What i am trying to do here, is to run a set of docker by chef docker cookbook version 2.14.3, with names looping into "localfolder", and when localfolder = "bootproxy", execute bash command to check what dockers are currently running and link them by a nginx docker named as "bootproxy".

My issue here is, whenever i cleared all my cache or after i have re-uploaded all my cookbook "webserver", chef-client runs fine without error. But when i rerun the chef-client, i got

Docker::Error::ServerError
--------------------------
Could not get container for bootproxy

due to the value of "links" in "docker_container" resources became "#" instead of an array that included the current running docker names like [ "container1:container1", "container2:container2", "container3:container3"]. So i suspect that the ruby_block that trying to get value from the host is being cached and not running after the first successive execution. My removal of cache (rm -rf /var/chef/cache) proved it but i cannot define removal of cache inside cookbook(not a neat way to work it out too). I need to make the chef-client able to rerun as i am using it to deploy my set of codes in the whole envionment. Please give me any advise for this.

Thanks!

Answer 1

I think you missed fixing the second resource, it isn't using the lazy{} helper.

Overall this code is kind of a mess so it's really hard to tell what it is doing in the first place. I recommend writing a lot of tests and maybe more comments.