简体繁体 English

比数据表更有效

[英]More efficient than a Datatable

原文 2011-06-20 15:22:49 1 3 c#/ performance

We have a reporting tool that is grabbing a large amount of records. 我们有一个报告工具，可以捕获大量记录。 At times it can be 1 million records. 有时可能是一百万条记录。 We have been storing this in a datable. 我们已经将其存储在数据库中。 I wanted to to know if there was a better object to store this in. I would need to be able to aggregate the data in various ways. 我想知道是否有更好的对象来存储它。我需要能够以各种方式聚合数据。

Update: Yes. 更新：是的。 Personally believe that should not being getting that many records. 个人认为不应获得那么多记录。 This is not the direction I want to go. 这不是我要走的方向。

Also I am using Oracle 我也在使用Oracle

Update Update 更新更新

Sorry for the delay, but there are always fire to put out here. 很抱歉耽搁，但是这里总有火扑灭。 The main issue was they were running out of memory and getting memory errors. 主要问题是它们的内存不足并出现内存错误。 They had issues with the datatable releasing from memory and also binding to a datagridview. 他们在从内存释放数据表以及绑定到datagridview时遇到问题。 I guess what I was looking for was a lighter weight object that wouldn't take as much space. 我想我要找的是重量较轻的物体，不会占用太多空间。

After thinking about a little more, it really doesn't make any sense to get that much data as diagonalbatman mentioned. 考虑了一点之后，获取对角线蝙蝠侠提到的大量数据真的没有任何意义。 furthermore if we have just a few people are using it with these issues. 此外，如果我们只有少数人正在使用这些问题。 How is it going to scale. 它将如何扩展。

Unfortunately, I have a boss that doesn't listen and an offshore team that is too much of a "yes sir" type attitude. 不幸的是，我的老板不听，而离岸团队则过于“是先生”式的态度。 They are serializing the raw data (as an XML file) and releasing the raw data Datatable which I think is not a good direction at all. 他们正在将原始数据序列化（作为XML文件）并发布原始数据Datatable，我认为这根本不是一个好方向。

@diagonalbatman - OUt of curiousity, do you have an example of this @diagonalbatman-好奇的你，你有这个例子吗

3 个解决方案

Why do you need to draw down 1 Milion records into your app? 为什么需要将100万条记录提取到您的应用程序中？

Can you not do your reporting consolidation / aggregation on the DB? 您不能在数据库上进行报告合并/聚合吗？ This would make better use of the DB's resources (after all this is what an RDBMS is designed to do) then you can focus your app on working with smaller consolidated sets? 这样可以更好地利用数据库的资源（毕竟这是RDBMS所要做的），然后您可以将应用程序集中在处理较小的合并集上吗？

I would recommend you try several options to verify, especially in light of your needed ability to aggregate the data in various ways. 我建议您尝试几种方法进行验证，尤其是考虑到您需要以各种方式汇总数据的能力。

1) Can it be aggregated by proper queries on the data side, this is likely the best solution. 1）可以通过在数据端进行适当的查询来汇总它，这可能是最好的解决方案。

2) if you use POCOs does LINQ improve upon your current memory and performance characteristics. 2）如果您使用POCO，则LINQ会改善您当前的内存和性能特征。 Does LINQ allow you to to the aggregation you require. LINQ是否允许您进行所需的聚合。

Measure the characteristics you care about and try different options. 测量您关心的特征，然后尝试其他选择。

What you want are Data Cubes. 您想要的是数据多维数据集。 Depending on the type of database you have, you should look at building some Cubes. 根据您拥有的数据库类型，您应该考虑构建一些多维数据集。