Clojure。 Http流文件

Question

我想流式傳輸大型二進制文件（exe、jpg、...，所有類型的文件）。 似乎aleph客戶端可以做到。 看了官方的sample，明白了如果我們把lazy序列傳給body，響應就可以傳一個流。

(defn streaming-numbers-handler
  "Returns a streamed HTTP response, consisting of newline-delimited numbers every 100
   milliseconds.  While this would typically be represented by a lazy sequence, instead we use
   a Manifold stream.  Similar to the use of the deferred above, this means we don't need
   to allocate a thread per-request.
   In this handler we're assuming the string value for `count` is a valid number.  If not,
   `Integer.parseInt()` will throw an exception, and we'll return a `500` status response
   with the stack trace.  If we wanted to be more precise in our status, we'd wrap the parsing
   code with a try/catch that returns a `400` status when the `count` is malformed.
   `manifold.stream/periodically` is similar to Clojure's `repeatedly`, except that it emits
   the value returned by the function at a fixed interval."
  [{:keys [params]}]
  (let [cnt (Integer/parseInt (get params "count" "0"))]
    {:status 200
     :headers {"content-type" "text/plain"}
     :body (let [sent (atom 0)]
             (->> (s/periodically 100 #(str (swap! sent inc) "\n"))
               (s/transform (take cnt))))}))

我有以下代碼：

(ns core
  (:require [aleph.http :as http]
            [byte-streams :as bs]
            [cheshire.core :refer [parse-string generate-string]]
            [clojure.core.async :as a]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [clojure.tools.logging :as log]
            [clojure.walk :as w]
            [compojure.core :as compojure :refer [ANY GET defroutes]]
            [compojure.route :as route]
            [me.raynes.fs :as fs]
            [ring.util.response :refer [response redirect content-type]]
            [ring.middleware.params :as params]
            [ring.util.codec :as cod])

  (:import java.io.File)
  (:import java.io.FileInputStream)
  (:import java.io.InputStream)
  (:gen-class))


(def share-dir "\\\\my-pc-network-path")

(def content-types
  {".png"  "image/png"
   ".GIF"  "image/gif"
   ".jpeg" "image/jpeg"
   ".svg"  "image/svg+xml"
   ".tiff" "image/tiff"
   ".ico"  "image/vnd.microsoft.icon"
   ".bmp"  "image/vnd.wap.wbmp"
   ".css"  "text/css"
   ".csv"  "text/csv"
   ".html" "text/html"
   ".txt"  "text/plain"
   ".xml"  "text/xml"})

(defn byte-seq [^InputStream is size]
  (let [ib (byte-array size)]
    ((fn step []
       (lazy-seq
         (let [n (.read is ib)]
           (when (not= -1 n)
             (let [cb (chunk-buffer size)]
               (dotimes [i size] (chunk-append cb (aget ib i)))
               (chunk-cons (chunk cb) (step))))))))))

(defn get-file [req]
  (let [uri (:uri req)
        file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
        full-dir (io/file share-dir file)
        _ (log/debug "FULL DIR: "full-dir)
        filename (.getName (File. (.getParent full-dir)))
        extension (fs/extension filename)
        _ (log/debug "EXTENSION: " extension)
        resp {:status  200
              :headers {"Content-type"        (get content-types (.toLowerCase extension) "application/octet-stream")
                        "Content-Disposition" (str "inline; filename=\"" filename "\"")}
              :body    (with-open [is (FileInputStream. full-dir)]
                         (let [bs (byte-seq is 4096)]
                           (byte-array bs)))}
        ]
    resp))

(def handler
  (params/wrap-params
    (compojure/routes
      (GET "/file/*"         [] get-file)
      (route/not-found "No such file."))))


(defn -main [& args]
  (println "Starting...")
  (http/start-server handler {:host "0.0.0.0" :port 5555}))

我得到了 uri 並嘗試使用塊讀取文件。 我想這樣做是因為文件可能有大約 3 GB。 所以，我期望應用程序不會使用比塊大小更多的內存。 但是當我為應用程序設置 1GB（-Xmx 選項）時，它占用了所有內存（1 GB）。 為什么要占用 1GB？ JVM 是這樣工作的嗎？ 當我有 100 個同時連接時（例如，每個文件是 3GB），我得到 OutOfMemoryError。

任務是使用塊“流式傳輸”文件以避免 OutOfMemoryError。

Answer 1

在get-file函數中，您正在對byte-seq的結果調用byte-array 。 byte-array將實現byte-seq返回的LazySeq ，這意味着所有這些都將在內存中。

據我所知（至少對於 TCP 服務器），aleph 接受 ByteBuffer 的任何惰性序列作為主體，並將為您最佳地處理它，因此您只需返回調用byte-seq的結果

(defn get-file [req]
  (let [uri (:uri req)
        file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
        full-dir (io/file share-dir file)
        _ (log/debug "FULL DIR: "full-dir)
        filename (.getName (File. (.getParent full-dir)))
        extension (fs/extension filename)
        _ (log/debug "EXTENSION: " extension)
        resp {:status  200
              :headers {"Content-type"        (get content-types (.toLowerCase extension) "application/octet-stream")
                        "Content-Disposition" (str "inline; filename=\"" filename "\"")}
              :body    (with-open [is (FileInputStream. full-dir)]
                         (byte-seq is 4096))}
        ]
    resp))

選擇

Aleph 符合環規范並且:body接受File或InputStream 。 所以沒有必要自己返回字節。

(defn get-file [req]
  (let [uri (:uri req)
        file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
        full-dir (io/file share-dir file)
        _ (log/debug "FULL DIR: "full-dir)
        filename (.getName (File. (.getParent full-dir)))
        extension (fs/extension filename)
        _ (log/debug "EXTENSION: " extension)
        resp {:status  200
              :headers {"Content-type"        (get content-types (.toLowerCase extension) "application/octet-stream")
                        "Content-Disposition" (str "inline; filename=\"" filename "\"")}
              :body    full-dir}
        ]
    resp))