How to read small chunks of data in phoenix controller, using Plug.Conn

Question

My goal is to be able to process a chunked HTTP request within a phoenix controller. I think the solution is to use Plug.Conn.read_body however I am getting an error or a timeout.

Currently I think the best solution is a custom parser.

defmodule Plug.EventStreamParser do
  @behaviour Plug.Parsers
  alias Plug.Conn

  def parse(conn, "text", "event-stream", _headers, _opts) do
    Conn.read_body(conn, length: 2, read_length: 1, read_timeout: 1_000)
    |> IO.inspect
    {:ok, %{}, conn}
  end
end

However I always get the {:error :timeout} at the inspect line.

Answer 1

Plug.Conn.read_body/2 only reads one chunk of the request body. You need to call it recursively in order to read everything. You also don't need to write a Parser to just read the body in chunks (I don't think a Parser can even do that if I understand your question correctly); if the Content-Type of the request is not one that Plug parses by default, you can call Plug.Conn.read_body/2 from your controller.

Here's a small implementation of recursively calling Plug.Conn.read_body/2 from a controller:

defmodule MyApp.PageController do
  use MyApp.Web, :controller

  def index(conn, _params) do
    {:ok, conn} = read_body(conn, [length: 1024, read_length: 1024], fn chunk ->
      # We just print the size of each chunk we receive.
      IO.inspect byte_size(chunk)
    end)
    text conn, "ok"
  end

  def read_body(conn, opts, f) do
    case Plug.Conn.read_body(conn, opts) do
      {:ok, binary, conn} ->
        f.(binary)
        {:ok, conn}
      {:more, binary, conn} ->
        f.(binary)
        read_body(conn, opts, f)
      {:error, term} ->
        {:error, term}
    end
  end
end

This reads the body in chunks of approximately 1024 bytes (it's not guaranteed that the binary returned is exactly the same size as requested). With the following request that POSTs 4000 bytes:

$ head -c 4000 /dev/urandom | curl -XPOST http://localhost:4000 --data-binary @- -H 'Content-Type: application/vnd.me.raw'
ok

The following is logged to the console:

[info] POST /
[debug] Processing by MyApp.PageController.index/2
  Parameters: %{}
  Pipelines: [:api]
1024
1024
1024
928
[info] Sent 200 in 3ms