每天处理10 B行数据以创建变量（计算列）的最佳方法是什么？

Question

Imagine you have a historical data and every day a couple of million rows of data gets added to it. There is a need to process the whole data on a daily basis and update variables. How would you approach this problem using Big data platform?

Happy to provide more details if needed.

Answer 1

Try very hard not to reprocess the whole 10B rows... I don't know what exactly you are looking for in that large of a dataset, but there is very likely a statistical model in which you can keep summary information, and just reprocess the incremental against that.

cricket_007 is right though, HDFS and Spark are likely your first tools of choice.