如何在Ruby中并行处理二进制文件？

Question

我正在尝试制作一个将二进制文件拆分为块并上传的函数

class ChunksClient < ApiStruct::Client
  # Takes the file, splits it into chunks and uploads each chunk into array of urls
  # in corresponding order
  def upload_chunks(big_file, array_of_urls)
    chunk_size = 5242880
    links.each do |link|
      chunk = object.read(chunk_size)
      upload_chunk(chunk, link)
    end
  end

  def upload_chunk(chunk, link)
    put(path: link, body: chunk, headers: { 'Content-type': 'application/octet-stream' })
  end
end

但是，一次做一大块很慢。 所以我尝试并行处理它们：

class ChunksClient < ApiStruct::Client
  # Takes the file, splits it into chunks and uploads each chunk into array of urls
  # in corresponding order
  def upload_chunks(big_file, array_of_urls)
    @chunk_size = 5242880
    @index = 0
    @object = object
    threads = []
    links.each do
      threads << Thread.new do
        chunk, index = take_chunk_with_index
        upload_chunk(chunk, links[index])
      end
    end
    threads.each(&:join)
  end

  private

  def upload_chunk(chunk, link)
    put(path: link, body: chunk, headers: { 'Content-type': 'application/octet-stream' })
  end

  def take_chunk_with_index
    index = @index
    chunk = @object.read(@chunk_size)
    @index += 1
    [chunk, index]
  end
end

但它每次都将块放入随机链接中。 我可以将块加载到内存中，但这样上传大文件（例如，以千兆字节为单位）会遇到问题

是否有使用线程处理二进制文件的正确方法？

Answer 1

您应该像这样将take_chunk_with_index方法与Mutex同步；

@mutex = Mutex.new

def take_chunk_with_index
  @mutex.synchronize do
    index = @index
    chunk = @object.read(@chunk_size)
    @index += 1
    [chunk, index]
  end
end