[英]Clojure core.async, CPU hangs after timeout. Anyway to properly kill macro thread produced by (go..) block?
Based on core.async walk through example , I created below similar code to handle some CPU intensive jobs using multiple channels with a timeout of 10 seconds. 基于core.async遍历示例 ,我在下面创建了类似的代码,以使用多个通道处理一些CPU密集型作业,超时为10秒。 However after the main thread returns, the CPU usage remains around 700% (8 CPUs machine). 但是在主线程返回后,CPU使用率仍然保持在700%左右(8台CPU机器)。 I have to manually run nrepl-close in emacs to shut down the Java process. 我必须在emacs中手动运行nrepl-close来关闭Java进程。
Is there any proper way to kill macro thread produced by (go..) block ? 是否有任何正确的方法来杀死(go ..)块产生的宏线程? I tried close! 我试过了! each chan, but it doesn't work. 每个陈,但它不起作用。 I want to make sure CPU usage back to 0 by Java process after main thread returns. 我希望在主线程返回后确保Java进程将CPU使用率恢复为0。
(defn [] RETURNED-STR-FROM-SOME-CPU-INTENSE-JOB (do... (str ...)))
(let [n 1000
cs (repeatedly n chan)]
(doseq [c cs]
(go
(>! c (RETURNED-STR-FROM-SOME-CPU-INTENSE-JOB ))))
(dotimes [i n]
(let [[result source] (alts!! (conj cs (timeout 10000))) ] ;;wait for 10 seconds for each job
(if (list-contains? cs source) ;;if returned chan belongs to cs
(prn "OK JOB FINISHED " result)
(prn "JOB TIMEOUT")
)))
(doseq [i cs]
(close! i)) ;;not useful for "killing" macro thread
(prn "JOBS ARE DONE"))
;;Btw list-contains? function is used to judge whether an element is in a list
;;http://stackoverflow.com/questions/3249334/test-whether-a-list-contains-a-specific-value-in-clojure
(defn list-contains? [coll value]
(let [s (seq coll)]
(if s
(if (= (first s) value) true (recur (rest s) value))
false)))
In REPL there seems to be no clean way yet. 在REPL中似乎没有干净的方式。
I first tried a very dirty way by using deprecated method Thread.stop 我首先使用弃用的方法Thread.stop尝试了一种非常脏的方法
(doseq [i @threadpool ]
(.stop i))
It seemed worked as CPU usage dropped once the main thread returned to REPL, but if I run the program again in REPL, it'd just hang at the go block part!! 一旦主线程返回到REPL,它似乎有效,因为CPU使用率下降了,但如果我在REPL中再次运行该程序,它只会挂在go块部分!
Then I googled around and found this blog and it says 然后我用Google搜索并发现了这个博客
One final thing to note: we don't explicitly do any work to shutdown the go routines. 最后要注意的一点是:我们没有明确地做任何工作来关闭go例程。 Go routines will automatically stop operation when the main function exits. 当主函数退出时,Go例程将自动停止操作。 Thus, go routines are like daemon threads in the JVM (well, except for the "thread" part ...) 因此,go例程就像JVM中的守护进程线程一样(除了“线程”部分...)
So I tried again by making my project into a uberjar and run it on a command console, and it turned out that CPU usage would drop immediately when blinking cursor returns to the console! 所以我再次尝试将我的项目变成一个uberjar并在命令控制台上运行它,结果当闪烁的光标返回到控制台时,CPU使用率会立即下降!
Based on answer for another related question How to control number of threads in (go...) , I've found a better way to properly kill all the threads started by (go...) block: 基于另一个相关问题的答案如何控制(go ...)中的线程数 ,我找到了一种更好的方法来正确杀死由(go ...)块启动的所有线程:
First alter the executor var and supply a custom thread pool 首先更改执行程序var并提供自定义线程池
;; def, not defonce, so that the executor can be re-defined
;; Number of threads are fixed to be 4
(def my-executor
(java.util.concurrent.Executors/newFixedThreadPool
4
(conc/counted-thread-factory "my-async-dispatch-%d" true)))
(alter-var-root #'clojure.core.async.impl.dispatch/executor
(constantly (delay (tp/thread-pool-executor my-executor))))
Then call .shutdownNow and .awaitTermination method at the end of (go...) block 然后在(go ...)块结束时调用.shutdownNow和.awaitTermination方法
(.shutdownNow my-executor)
(while (not (.awaitTermination my-executor 10 java.util.concurrent.TimeUnit/SECONDS ) )
(prn "...waiting 10 secs for executor pool to finish") )
[UPDATE] The shutdown executor method above seems not pure enough. [UPDATE]上面的关闭执行程序方法似乎不够纯粹。 The final solution for my case is to send a function with control of its own timeout into go block, using thunk-timeout function. 我的最终解决方案是使用thunk-timeout函数将控制其自身超时的函数发送到go块。 Credits go to this post . 积分转到这篇文章 。 Example below 以下示例
(defn toSendToGo [args timeoutUnits]
(let [result (atom nil)
timeout? (atom false)]
(try
( thunk-timeout
(fn [] (reset! result (myFunction args))) timeoutUnits)
(catch java.util.concurrent.TimeoutException e (do (prn "!Time out after " timeoutUnits " seconds!!") (reset! timeout? true)) ))
(if @timeout? (do sth))
@result))
(let [c ( chan)]
(go (>! c (toSendToGo args timeoutUnits))))
(shutdown-agents)
Implementation-specific, JVM: both agents and channels use a global thread pool, and the termination function for agents iterates and closes all open threads in the VM. 特定于实现的JVM: 代理和通道都使用全局线程池,代理的终止功能迭代并关闭VM中的所有开放线程。 Empty the channels first : this action is immediate and non-reversible (especially if you are in a REPL). 首先清空通道 :此操作是立即且不可逆的(特别是如果您处于REPL中)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.