[英]Extract result from Spark Stream into Java Object
Currently I have integrated Spark Stream with Kafka in Java and able to aggregate the stats. 目前,我已经在Java中将Spark Stream与Kafka集成在一起,并且能够汇总统计信息。 However, I cannot figure out a way to store the result into a Java object so I can pass this object with the result around with different methods/classes without storing them into database.
但是,我无法找到一种将结果存储到Java对象中的方法,因此我可以使用不同的方法/类将结果与该对象一起传递,而无需将其存储到数据库中。 I have spent quite amount of time searching for tutorial/examples online but all of them are end up with using print() to display the result in console.
我花了很多时间在网上搜索教程/示例,但是所有这些最终都使用print()在控制台中显示了结果。 However, what I am trying to do is to return these results JSON string when users call a rest-api endpoint.
但是,我试图做的是当用户调用rest-api端点时返回这些结果JSON字符串。
Is it possible that I can have these results in memory and pass them around with different methods, or I need to store them into database first and fetch them from there as needed? 是否可以将这些结果存储在内存中并以不同的方法传递给它们,或者我需要先将它们存储到数据库中,然后根据需要从那里获取它们?
If I got you right you want consume your results from Spark Streaming via Rest APIs. 如果我答对了,您想通过Rest API消耗Spark Streaming的结果。
Even if there are some ways to directly accomplish this (eg using Spark SQL/Thrift server) I would separate these two tasks. 即使有一些方法可以直接完成此任务(例如,使用Spark SQL / Thrift服务器),我也会将这两个任务分开。 Else if you're Spark Streaming process fails, your service/REST-API layer will fail too.
否则,如果您的Spark Streaming过程失败,那么您的服务/ REST-API层也会失败。
Thus it has its advantages to separate these two layers. 因此,将这两层分开具有其优点。 You are not forced to use a classical database.
您没有被迫使用经典数据库。 You could implement a service, which implements/uses JCache and send your results of the Spark streaming process to it.
您可以实现一个服务,该服务实现/使用JCache并将Spark流处理的结果发送给它。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.