无需模板引擎即将数据从数据库发送到客户端（node.js）
[英]Send data from database to client without templating engine (node.js)
问题描述
Hi I was running some test with Node.js vs Fulephp. 
嗨，我正在使用Node.js和Fulephp进行一些测试。 I had a simple set up and I was trying to see what is faster. 我有一个简单的设置，我试图看到更快。 I had 10 thousand records in mogodb being pulled in to views. 我在mogodb中有1万条记录被录入视图。 The set up was simple, no js, no css, with minimal html. 设置很简单，没有js，没有css，只有最小的html。 I quickly noticed that php set up was twice as fast. 我很快注意到php的设置速度是原来的两倍。 At first I dismissed nodejs as being slower and moved on with my life. 起初我认为nodejs比较慢，并继续我的生活。 However I decided to try node without jade which I used as my templating engine, and by stroke of luck I came across a post on stackoverflow that philosophy behind jade is not so much speed but elegance. 然而，我决定尝试没有玉的节点，我用它作为我的模板引擎，幸运的是，我遇到了一个关于stackoverflow的帖子，玉背后的哲学不是速度而是优雅。 Then I decided to try node without any temp. 然后我决定尝试没有任何温度的节点。 engines. 引擎。 But I quickly ran into a problem since I realized that I have no idea how to pass data from database and node to client . 但是我很快就遇到了问题，因为我意识到我不知道如何将数据从数据库和节点传递到客户端 。 I was in for a long night of horror and despair. 我在一个漫长的恐怖和绝望的夜晚。 At some point I came to a conclusion that I need socket.io's help. 在某些时候，我得出结论，我需要socket.io的帮助。 Though I was able to connect to socket.io eventually I still was not able to figure out how to pass the data. 虽然我能够连接到socket.io最终我仍然无法弄清楚如何传递数据。 Then I decided to go back to using an temp. 然后我决定回去使用临时工。 engine, but this time I decided to try ejs. 引擎，但这次我决定尝试ejs。 Eventually I was able to render some data which had the following form [object Object] , but it was not 10 thousand records, more like 25. I decided to do the right thing and post my question here. 最终我能够呈现一些具有以下形式[object object]的数据 ，但它不是一万条记录，更像是25条。我决定做正确的事情并在这里发布我的问题。 I would like to render view without templating engine to see if my assumptions are right. 我想在没有模板引擎的情况下渲染视图，看看我的假设是否正确。 After all I am trying to do two things understand how to pass data to the client form node.js and see if it will improve my performance. 毕竟我正在尝试做两件事，了解如何将数据传递到客户端表单node.js，看看它是否会提高我的性能。
Here is my app.js with some comments: 这是我的app.js，有一些评论：
/**
* Mongo DB
*/
var mongous = require('mongous').Mongous,
dbCollection = 'test.personnel';
/**
* Module dependencies.
*/
var express = require('express'),
app = module.exports = express.createServer(),
pub = __dirname + '/public';
// Configuration
app.configure(function(){
app.set('view options', {layout: false});
//not sure if i need these here, but left it in case
app.use(app.router);
app.use(express.static(pub));
});
//Simple templating
//I took this example from stackoverflow,
//can't find that post anymore,
//though I can look if need be
app.register('.html', {
compile: function(str, options){
return function(locals){
return str;
}
}
});
// Routes
//This is where data is right now, it need to end up on the client side
//This was working with jade and kinda worked with ejs (though I am not sure because I was getting [object Object])
//--------------------------------------------------
app.get('/', function(req, res){
mongous(dbCollection).find(function(output){
res.render('index.html',{
//the data is here, it works, i tested it with console.log
data: output.documents
});
});
});
//--------------------------------------------------
app.configure('production', function(){
app.use(express.errorHandler());
});
app.listen(3000);
console.log('Express server listening on port %d in %s mode', app.address().port, app.settings.env);
and here is my view: 这是我的观点：
<!DOCTYPE html>
<html>
<head>
<meta charset='utf-8'>
<title>Index</title>
</head>
<body>
<div id="data">
Data needs to end up here. Somehow...
</div>
</body>
</html>
As you can see not much in there. 你在那里看不到多少。 Now I understand that I will most likely will need to use some sort of templating engine on the client side and once I have the data on client side, I will be able to work it out myself. 现在我明白我很可能需要在客户端使用某种模板引擎，一旦我在客户端获得数据，我就能自己解决。 Which maybe even slower in the end, but my primary goal is to understand how to pass data in node.js to client so that I could continue experimenting. 最后可能更慢，但我的主要目标是了解如何将node.js中的数据传递给客户端，以便我可以继续进行实验。 Please help if you can, it will improve my understanding of node significantly. 请尽可能帮助，这将显着提高我对节点的理解。 Thank you. 谢谢。
EDIT: With the help of all of you and especially josh3736 this is what I ended up with, if you are interested... http://pastie.org/private/z3fjjbjff8284pr2mafw 编辑：在你们所有人的帮助下，尤其是josh3736 ，如果你有兴趣的话，这就是我最终的结果...... http://pastie.org/private/z3fjjbjff8284pr2mafw
4 个解决方案
解决方案1
5 已采纳 2012-02-07 04:48:40
As you allude to in your answer, part of the problem is the speed of the templating engine itself; 正如你在答案中提到的，部分问题是模板引擎本身的速度; you've found out that Jade is not the fastest — in fact, it's one of the slowest . 你发现玉不是最快的 - 事实上， 它是最慢的之一 。
My favorite engine is doT . 我最喜欢的引擎是doT 。 In the performance test I linked to, doT can render the template 5.4 million times per second. 在我链接的性能测试中，doT可以每秒渲染模板540 万次。 Jade can render a similar template only 29,000 times per second. Jade每秒只能渲染29,000次类似的模板。 It's not even a contest. 这甚至不是一场比赛。
However, templating engine speed is only a small part of the issue here. 然而，模板引擎速度只是这里问题的一小部分。 I believe your real problem is the Mongo driver you're using appears to be poorly designed for Node's asynchronous model. 我相信你真正的问题是你正在使用的Mongo驱动程序似乎设计不好用于Node的异步模型。 (Disclaimer: I've never actually used Mongous; I just spent a few minutes looking over the code.) （免责声明：我从未真正使用Mongous;我只花了几分钟查看代码。）
Node is meant to work with streams of data. 节点旨在处理数据流。 In other words, you're supposed to operating on very small chunks of data at a time. 换句话说，您应该一次在非常小的数据块上运行。 In contrast, it looks like Mongous processes the entire dataset and returns it to your code as one JSON object. 相比之下，看起来像Mongous处理整个数据集并将其作为一个JSON对象返回到您的代码。
This is convenient and fine for small datasets, but completely falls apart when working with large amounts of data like you are (10,000 records). 这对于小型数据集来说非常方便，但是在处理大量数据（10,000条记录）时会完全崩溃。 Node will be completely locked up while parsing and handling that much data (which is very, very bad since it won't be able to handle any incoming connections), and the V8 memory management system isn't optimized for large heap allocations like that. 在解析和处理那么多数据时，节点将被完全锁定（这非常非常糟糕，因为它无法处理任何传入连接），并且V8内存管理系统未针对大堆分配进行优化。
To work with large datasets properly, you have to use a Mongo driver that streams records to your code, like node-mongodb-native, or mongoskin , which makes the API a little easier to deal with. 要正确使用大型数据集，您必须使用Mongo驱动程序将记录流式传输到您的代码，例如node-mongodb-native或mongoskin ，这使得API更容易处理。
maerics' answer was on the right track, but is wrong because it uses toArray , which creates the same problem you have under Mongous: the entire dataset is collated into an in-memory array. maerics的答案是在正确的轨道上，但是错误是因为它使用了toArray ，它会在Mongous下产生同样的问题： 整个数据集被整理成一个内存数组。 Instead, just use the cursor's each method, which asynchronously calls your callback function for each returned record as it comes in . 相反，只需使用鼠标的each方法，它异步调用的，因为它涉及每一个返回的记录你的回调函数。 This way, the entire resultset is never in memory; 这样，整个结果集永远不会在内存中; you only work with one at a time, allowing the records you've already processed to be discarded and garbage collected if necessary. 您只能一次使用一个，允许丢弃您已处理的记录并在必要时收集垃圾。
Now that we've established how to get your data out of the database, we need to figure out how to get it to the client. 现在我们已经建立了如何从数据库中获取数据的方法，我们需要弄清楚如何将数据传输到客户端。
The problem here is that an Express' view system expects you to have all your data available up-front so that the templating engine can render out a single string to be sent to the client. 这里的问题是Express的视图系统要求您预先提供所有数据，以便模板引擎可以呈现出要发送给客户端的单个字符串。 As we discussed above, this isn't such a good idea if you're dealing with thousands of records. 正如我们上面所讨论的，如果您处理数千条记录，这不是一个好主意。 The proper way to do this is to stream the data we get from Mongo directly to the client. 正确的方法是将我们从Mongo直接获取的数据流式传输到客户端。 Unfortunately, we can't really do this within an Express view — they're not designed to be asynchronous. 不幸的是，我们无法在Express视图中真正做到这一点 - 它们并非设计为异步。
Instead, you're going to have to write a custom handler. 相反，您将不得不编写自定义处理程序。 You're already on that path with Hippo's answer and your own attempt, but what you really need to use is res.write() , not res.send . 你已经在Hippo的答案和你自己的尝试上走了这条路，但你真正需要使用的是res.write() ，而不是res.send 。 Like res.render , res.send expects you to have a complete response when you call it because it internally calls res.end , ending the HTTP response. 和res.render一样， res.send希望你在调用它时有一个完整的响应，因为它在内部调用res.end ，结束HTTP响应。 In contrast, res.write simply sends data over the network, leaving the HTTP response open and ready to send more data — in other words, streaming your response. 相比之下， res.write只是通过网络发送数据，使HTTP响应保持打开并准备发送更多数据 - 换句话说，流式传输您的响应。 (Keep in mind that you have to set any HTTP headers before you start streaming. For example, res.contentType('text/html'); ) （请记住，在开始流式传输之前必须设置任何HTTP标头。例如， res.contentType('text/html'); ）
Just because you're manually handling the response (foregoing the view rendering system) does not preclude you from taking advantage of a templating engine. 仅仅因为您手动处理响应（在视图渲染系统之前）并不妨碍您利用模板引擎。 You can use a template for the header and footer of your document and one for each record. 您可以使用模板作为文档的页眉和页脚，每个记录使用一个模板。 Let's use doT to put everything together. 让我们用doT把所有东西放在一起。
First, let's declare our templates. 首先，让我们声明我们的模板。 In real life, you might load these from files (or even hack Express to load them for you as views and get the template source), but we'll just declare them in our code. 在现实生活中，你可能会从文件加载这些（甚至破解快速加载它们对你的看法，并获得模板源），但我们只是宣布他们在我们的代码。
var header = doT.template('<!DOCTYPE html><html><head><title>{{=it.title}}</title></head><body>');
var record = doT.template('<p>{{=it.fieldname}}, ...</p>');
var footer = doT.template('</body></html>');
( doT.template returns a function which generates HTML from the template we gave above and an object you pass to that returned function when you invoke it. For example, now we can call header({title:'Test'}); ) （ doT.template返回一个函数，该函数从我们上面给出的模板生成HTML，以及在调用它时传递给返回函数的对象。例如，现在我们可以调用header({title:'Test'}); ）
Now we do the real work in the request handler: (assumes we already have a collection from the Mongo driver) 现在我们在请求处理程序中完成实际操作:(假设我们已经拥有Mongo驱动程序的collection ）
app.get('/', function(req, res){
res.contentType('text/html');
res.write(header({title:'Test'}));
collection.find({}).each(function(err, doc) {
if (err) return res.end('error! ' + err); // bail if there's an error
if (doc) {
res.write(record(doc));
} else { // `doc` will be null if we've reached the end of the resultset.
res.end(footer());
}
});
});
解决方案2
0 2012-02-05 23:22:15
If you're interested in a really raw performance test between PHP and Node.js then you should consider ditching all the middleware on the Node stack (eg Express.js, and Mongous) for the direct approach using a MongoDB native driver and raw HTML generation. 如果您对PHP和Node.js之间真正原始的性能测试感兴趣，那么您应该考虑使用MongoDB本机驱动程序和原始HTML放弃Node堆栈上的所有中间件（例如Express.js和Mongous）。代。 Something like this (untested): 像这样（未经测试）：
var http = require('http')
, mongo = require('mongodb')
, db = new mongo.Db('foo', new mongo.Server('localhost', 27017, {}));
db.open(function(err, db) {
if (err) throw err;
db.collection('people', function(err, collection) {
var server = http.createServer(function(req, res) {
res.writeHead('200', {'Content-Type': 'text/html'});
res.write('<html><head><title>Node.js Output</title></head>');
res.write('<body><div id="data">');
collection.find(function(err, cursor) {
cursor.each(function(err, doc) {
if (doc) {
res.write('<div>' + doc.name + '</div>');
} else {
res.end('</div></body></html>');
cursor.close();
}
});
});
}).listen(8080, 'localhost');
console.log('OK: listening on http://localhost:8080/');
});
});
[Edit] Updated the use of the mongo driver to use a cursor instead of the "toArray" method. [编辑]更新了mongo驱动程序的使用，以使用游标而不是“toArray”方法。
A key choice here is the Node MongoDB driver; 这里的关键选择是Node MongoDB驱动程序; in particular, you want one that is as non-blocking as possible. 特别是，你想要一个尽可能无阻塞的。 Surprisingly, there doesn't seem to be a standout one right now (IMHO). 令人惊讶的是，现在似乎没有一个出色的（恕我直言）。
解决方案3
0 2012-02-05 23:30:23
If you stick to express and some kind of database layer, use res.send() instead of res.render . 如果您坚持使用express和某种数据库层，请使用res.send()而不是res.render 。
Also keep in my mind that node.js is quite new, so not every library is stable or fast as you are used from other languages. 另外请记住，node.js是新的，因此并不是每个库都是稳定的或快速的，因为你使用的是其他语言。 (Eg: You might be faster using another approach for accessing mongo, or not using express.) （例如：您可能更快地使用另一种方法来访问mongo，或者不使用express。）
解决方案4
0 2012-02-07 03:06:40
Well this is best what I was able to come up with. 嗯，这是我能想到的最好的。 It is of course based on Hippos's answer but with a little bit more stuff. 它当然是基于Hippos的答案，但有更多的东西。 However, now I am doing something really wrong, because now it seems to be even slower than with templateing engine, much slower ... 但是，现在我正在做一些非常错误的事情，因为现在它似乎比使用模板引擎更慢 ， 慢得多 ......
The code in question: 有问题的代码：
//--------------------------------------------------
app.get('/', function(req, res){
mongous(dbCollection).find(10000,function(output){
var o = output.documents,
str = JSON.stringify(o);
res.send(str);
});
});
//--------------------------------------------------
Sigh... 叹...
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.