[英]Clojure async/go how to park blocking code
I use some Java library that makes non-async get and post requests. 我使用一些Java库来进行非异步get和post请求。 I used to wrap such requests to futures and it solves for me the "waiting problem" (I mean waiting for the response)
我曾经把这些请求包装到期货中,它解决了我的“等待问题”(我的意思是等待响应)
(defn unchangeable-lib-request [n]
(Thread/sleep 1000)
n)
(defn process [n]
(let [res (atom [])]
(dotimes [i n]
(future (swap! res conj (unchangeable-lib-request i))))
(loop []
(if (> n (count @res))
(recur)
@res))))
(time (process 9))
;; "Elapsed time: 1000.639079 msecs"
;; => [8 7 5 6 4 3 2 1 0]
But I need to create hundreds of requests and this creates performance problems. 但我需要创建数百个请求,这会产生性能问题。 I found out about core.async and go blocks.
我发现了core.async和go块。 But if I will use go-blocks with this library, it will not solve the "waiting problem"
但是如果我在这个库中使用go-blocks,它将无法解决“等待问题”
(defn unchangeable-lib-request [n]
(Thread/sleep 1000)
n)
(defn process [n]
(let [c (async/chan 10)]
(dotimes [i n]
(async/go
(async/>! c (unchangeable-lib-request i))))
(loop [result []]
(if (> n (count result))
(recur (conj result (async/<!! c)))
result))))
(time (process 9))
;; "Elapsed time: 2001.770183 msecs"
;; => [0 4 1 6 7 2 5 3 8]
Go blocks can handle just 8 requests simultaneously. Go块可以同时处理8个请求。 Is there a possibility to write some async-wrapper that will park go-block and provide ability to make 100s of requests asynchronously without blocking each other?
是否有可能编写一些async-wrapper来停止go-block并提供异步生成100个请求而不会相互阻塞的能力?
(defn process [n]
(let [c (async/chan 10)]
(dotimes [i n]
(async/go
(async/>! c (magic-async-parking-wrapper
(unchangeable-lib-request i))))
(loop [result []]
(if (> n (count result))
(recur (conj result (async/<!! c)))
result))))
(time (process 9))
;; "Elapsed time: 1003.2563 msecs"
I know about async/thread but it seems that this is the same as (future ...). 我知道异步/线程,但似乎这与(future ...)相同。
Is it possible? 可能吗?
I'd suggest: 我建议:
put!
put!
, something like: (future (put! chan (worker-function)))
(future (put! chan (worker-function)))
This is where you use clojure.core.async/pipeline-blocking
这是您使用
clojure.core.async/pipeline-blocking
(require '[clojure.core.async :as a :refer [chan pipeline-blocking]])
(let [output-chan (chan 100)
input-chan (chan 1000)]
(pipeline-blocking 4 ; parallelism knob
output-chan
(map unchangeable-lib-request)
input-chan)
;; Consume results from output-chan, put operations on input-chan
[output-chan input-chan]
)
This spawns n (in this case 4) threads that are kept busy executing unchangeable-lib-request
. 这会产生n(在这种情况下为4)线程,这些线程一直忙于执行
unchangeable-lib-request
。
Use the buffer size of output-chan
to finetune how much requests you want to happen in advance. 使用
output-chan
的缓冲区大小来提前调整您想要发生的请求数量。
Use the buffer size of input-chan
to finetune how many requests you want scheduled without backpropagation (a blocking input-chan
). 使用
input-chan
的缓冲区大小来微调您希望在没有反向传播的情况下安排的请求数量(阻塞input-chan
)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.