S3 Read Progress with Fog

Question

I am trying to download a large (>500mb) file from s3 with fog. I would like to check on progress, but it seems to download the file multiple times.

I'm using:

connction = Fog::Storage.new(
  provider: 'AWS',
  aws_access_key_id: key_id,
  aws_secret_access_key: secret_key,
  region: 'eu-west-1',
)

connection.get_object(bucket, obj_key) do |chunk, remaining_bytes, total_bytes|
  file.write(chunk)
  p "Writing file: #{remaining_bytes} / #{total_bytes} remaining"
end

And it seems to do the entire process multiple times, almost downloading all the data but when there's one megabyte left, it starts downloading all over again!

After roughly 4 loops it raises an error:

negative argument (ArgumentError) (Excon::Errors::SocketError)

Any advice greatly appreciated!

UPDATE: Gem versions:

excon (0.20.1)
    fog (1.10.1)
    builder
    excon (~> 0.20)
    formatador (~> 0.2.0)
    mime-types
    multi_json (~> 1.0)
    net-scp (~> 1.1)
    net-ssh (>= 2.1.3)
    nokogiri (~> 1.5.0)
    ruby-hmac

Answer 1

I wonder if this is an issue with the Excon gem. What version of Excon, Fog, and Ruby are you using?

You might also want to try re-running your script with EXCON_DEBUG=true. Hopefully you will get some better information.