[英]Highland JS - turn reduce result array back to stream
I'm experimenting with data streams where I would like to aggregate time series data.我正在尝试聚合时间序列数据的数据流。 The reduce works, however I cannot find a solution to turn back the resulted array to another stream.
减少工作,但是我找不到将结果数组转回另一个 stream 的解决方案。
When I call a map on the reduce I only get back the whole array as the the result.当我在 reduce 上调用 map 时,我只返回整个数组作为结果。 Not the data within the array.
不是数组中的数据。
Any thoughts or hints are welcome.欢迎任何想法或提示。
const fs = require('fs')
const highland = require('highland')
const streamAgg = (aggData, parts) => {
if (!aggData[parts.groupBySec]) {
aggData[parts.groupBySec] = {}
aggData[parts.groupBySec]['volume'] = parts.volume
aggData[parts.groupBySec]['start-time'] = parts.timeStamp
aggData[parts.groupBySec]['end-time'] = parts.timeStamp
} else {
aggData[parts.groupBySec]['volume'] += parts.volume
aggData[parts.groupBySec]['end-time'] = parts.timeStamp
}
return aggData
}
highland(fs.createReadStream('./timeseriesdata.csv', 'utf8'))
.split()
.map(line => line.split(','))
.map(parts => ({
timeStamp: parts[0],
timeStampParsed: Date.parse(parts[0]),
groupBySec: Math.floor(Date.parse(parts[0])/1000)*1000,
volume: Number(parts[3]),
}))
.reject(parts => isNaN(parts.timeStampParsed))
.reduce([], streamAgg)
.map(x => x)
.each(x => console.log(x))
Seems like you're reducing to an array instead of an object (what your reducer function seems to expect).好像你正在减少到一个数组而不是一个 object (你的减速器 function 似乎期望什么)。 This works:
这有效:
const fs = require('fs');
const highland = require('highland');
const streamAgg = (aggData, parts) => {
if (!aggData[parts.groupBySec]) {
aggData[parts.groupBySec] = {};
aggData[parts.groupBySec].volume = parts.volume;
aggData[parts.groupBySec]['start-time'] = parts.timeStamp;
aggData[parts.groupBySec]['end-time'] = parts.timeStamp;
} else {
aggData[parts.groupBySec].volume += parts.volume;
aggData[parts.groupBySec]['end-time'] = parts.timeStamp;
}
return aggData;
};
highland(fs.createReadStream('./timeseriesdata.csv', 'utf8'))
.split()
.map(line => line.split(','))
.map(parts => ({
timeStamp: parts[0],
timeStampParsed: Date.parse(parts[0]),
groupBySec: Math.floor(Date.parse(parts[0]) / 1000) * 1000,
volume: Number(parts[3])
}))
.reject(parts => isNaN(parts.timeStampParsed))
.reduce({}, streamAgg)
.doto(console.log)
.done(() => {
process.exit(0);
});
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.