[英]Processing a stream of messages from a http server in clojure
I am looking for an idiomatic way to do the following. 我正在寻找一种惯用的方法来做到以下几点。 I have a http server that on a particular GET request responds with a stream of messages.
我有一个http服务器,在特定的GET请求上响应消息流。 Now, since this message is non-terminating, when I use clj-http/get, the call just blocks forever (I am using LightTable).
现在,由于此消息是非终止的,当我使用clj-http / get时,调用会永远阻塞(我正在使用LightTable)。 I would like to set up a callback or a core.async style channel to do some operation on the message as it comes in. Even writing the stream to a file would be a good first step for me.
我想建立一个回调或core.async样式的通道来对消息进行一些操作。即使将流写入文件对我来说也是一个很好的第一步。 Any pointers?
有什么指针吗? Here is the call:
这是电话:
(require '[clj-http.client :as client])
(def url "http://hopey.netfonds.no/tradedump.php?date=20150508&paper=AAPL.O&csv_format=txt")
(client/get url)
The date has to be changed to today's date for the data to stream. 日期必须更改为今天的数据流日期。 Thanks!
谢谢!
To just write the stream to file, a simple approach is using clojure.java.io/copy
(which takes an input-stream such as that returned by (:body (client/get some-url {:as :stream}))
and an output stream and copies from one to the other). 要将流写入文件,一个简单的方法是使用
clojure.java.io/copy
(它接受输入流,例如(:body (client/get some-url {:as :stream}))
返回的输入流(:body (client/get some-url {:as :stream}))
和输出流以及从一个到另一个的副本)。 Something like 就像是
(ns http-stream
(:require [clj-http.client :as client]
[clojure.java.io :as io]))
(with-open [in-stream (:body (client/get "http://hopey.netfonds.no/tradedump.php?date=20150508&paper=AAPL.O&csv_format=txt" {:as :stream}))
out-stream (->> "streamoutput.txt"
io/as-file
io/output-stream)]
(io/copy in-stream out-stream))
That gave me several thousand lines of tab separated values over a couple seconds. 这让我在几秒钟内获得了数千行标签分隔值。 Now, to process them with core.async at the level of lines we probably want to process the stream a bit more using a
reader
and a line-seq
: 现在,要在行级别使用core.async处理它们,我们可能希望使用
reader
和line-seq
更多地处理流:
(ns http-stream
(:require [clj-http.client :as client]
[clojure.core.async :as async]
[clojure.java.io :as io]
[clojure.string :as str]))
(defn trades-chan
"Open the URL as a stream of trades information. Return a channel of the trades, represented as strings."
[dump-url]
(let[lines (-> dump-url
(client/get {:as :stream})
:body
io/reader
line-seq) ];;A lazy seq of each line in the stream.
(async/to-chan lines))) ;;Return a channel which outputs the lines
;;Example: Print the first 250 lines.
(let [a (trades-chan "http://hopey.netfonds.no/tradedump.php?date=20150508&paper=AAPL.O&csv_format=txt")]
(async/go-loop [takes 250]
(when (< 0 takes)
(println (async/<! a))
(recur (dec takes)))))
Now, with this you are largely started up, but I notice that the stream always starts with a description of what the columns are 现在,有了这个,你很大程度上已经启动了,但是我注意到流总是以对列的描述开始
time price quantity board source buyer seller initiator
and you can use that as a chance to improve a little bit. 你可以用它作为改善一点点的机会。 In particular, that's enough information to build a transducer for the trades-chan that can turn the trades into a more convenient format to work with, like a map.
特别是,这足以为交易建立一个传感器的信息,可以将交易变成更方便的格式,如地图。 Also, we likely want a way to stop taking elements and close the connection sometime.
此外,我们可能想要一种方法来停止使用元素并在某个时候关闭连接。 I'm not that familiar with core.async myself but this seems to work:
我自己并不熟悉core.async,但这似乎有效:
(defn trades-chan
"Open the URL as a tab-separated values stream of trades.
Returns a core.async channel of the trades, represented as maps.
Closes the HTTP stream on channel close!"
[dump-url]
(let[stream (-> dump-url
(client/get {:as :stream})
:body)
lines (-> stream
io/reader
line-seq) ;;A lazy seq of each line in the stream.
fields (map keyword (str/split (first lines) #"\t")) ;; (:time :price :quantity ...
transducer (map (comp #(zipmap fields %) #(str/split % #"\t"))) ;;A transducer that splits strings on tab and makes them into maps with keys from fields
output-chan (async/chan 50 transducer)]
(async/go-loop [my-lines (drop 1 lines)]
(if (async/>! output-chan (first my-lines)) ;;If we managed to put
(recur (rest my-lines)) ;;then the chan is not closed. Recur with the rest of the lines.
(.close stream))) ;;else close the HTTP stream.
output-chan))
I think user1571406's answer is reasonable and gives a good introduction to combining clj-http
with core.async
. 我认为user1571406的答案是合理的,并且给出了将
clj-http
与core.async
结合起来的一个很好的介绍。 However, if you do not stick to clj-http
, I would like to strongly recommend the http-kit library, which is more designed for asynchronous response handling. 但是,如果您不坚持使用
clj-http
,我强烈推荐使用http-kit库,它更适用于异步响应处理。 Using http-kit
, you can write your call back as follows. 使用
http-kit
,您可以按如下方式编写回叫。
user> (require '[clojure.java.io :as io]
'[org.httpkit.client :as h])
nil
user> (def url "http://hopey.netfonds.no/tradedump.php?date=20150508&paper=AAPL.O&csv_format=txt")
#'user/url
user> (h/get url {:as :stream}
(fn [{:keys [status body]}]
(if (= status 200)
(with-open [out (io/output-stream "/tmp/output.txt")]
(io/copy body out)))))
#<core$promise$reify__6363@373b22df: :pending>
The last h/get
function call returns immediately, and its callback fn
writes the body of the response to the file /tmp/output.txt
asynchronously. 最后一个
h/get
函数调用立即返回,其回调函数fn
异步将响应体写入文件/tmp/output.txt
。
(ns asyncfun.core
(:require [clojure.core.async :as async
:refer [<! >!! go chan]]
[clj-http.client :as client]))
(def url "http://hopey.netfonds.no/tradedump.php?date=20150508&paper=AAPL.O&csv_format=txt")
(def out-chan (chan))
(go (println (<! out-chan)))
(>!! out-chan (client/get url))
I put this code together in a couple minutes. 我把这段代码放在一起几分钟。 I think core.async is what you are looking for.
我认为core.async就是你要找的东西。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.