[英]Sending Streaming Data as JSON in Java/Scala
I'm used to python and using the Scala Spark Streaming libraries to handle real-time Twitter streaming data. 我习惯使用python,并使用Scala Spark流媒体库处理实时Twitter流数据。 Right now, I'm able to send as a string, however, my streaming service requires JSON.
现在,我可以作为字符串发送,但是我的流服务需要JSON。 Is there a way I can easily adapt my code to send as JSON dictionary instead of a String?
有没有一种方法可以使我的代码轻松适应以JSON字典而不是String的形式发送?
%scala
import scala.collection.JavaConverters._
import com.microsoft.azure.eventhubs._
import java.util.concurrent._
val namespaceName = "hubnamespace"
val eventHubName = "hubname"
val sasKeyName = "RootManageSharedAccessKey"
val sasKey = "key"
val connStr = new ConnectionStringBuilder()
.setNamespaceName(namespaceName)
.setEventHubName(eventHubName)
.setSasKeyName(sasKeyName)
.setSasKey(sasKey)
val pool = Executors.newFixedThreadPool(1)
val eventHubClient = EventHubClient.create(connStr.toString(), pool)
def sendEvent(message: String) = {
val messageData = EventData.create(message.getBytes("UTF-8"))
// CONVERT IT HERE?
eventHubClient.get().send(messageData)
System.out.println("Sent event: " + message + "\n")
}
import twitter4j._
import twitter4j.TwitterFactory
import twitter4j.Twitter
import twitter4j.conf.ConfigurationBuilder
val twitterConsumerKey = "key"
val twitterConsumerSecret = "key"
val twitterOauthAccessToken = "key"
val twitterOauthTokenSecret = "key"
val cb = new ConfigurationBuilder()
cb.setDebugEnabled(true)
.setOAuthConsumerKey(twitterConsumerKey)
.setOAuthConsumerSecret(twitterConsumerSecret)
.setOAuthAccessToken(twitterOauthAccessToken)
.setOAuthAccessTokenSecret(twitterOauthTokenSecret)
val twitterFactory = new TwitterFactory(cb.build())
val twitter = twitterFactory.getInstance()
val query = new Query(" #happynewyear ")
query.setCount(100)
query.lang("en")
var finished = false
while (!finished) {
val result = twitter.search(query)
val statuses = result.getTweets()
var lowestStatusId = Long.MaxValue
for (status <- statuses.asScala) {
if(!status.isRetweet()){
sendEvent(status.getText())
}
lowestStatusId = Math.min(status.getId(), lowestStatusId)
Thread.sleep(2000)
}
query.setMaxId(lowestStatusId - 1)
}
eventHubClient.get().close()
Scala has no native way to convert string to Json, you'll need to use an external library. Scala没有将字符串转换为Json的本地方法,您需要使用一个外部库。 I recommend using
Jackson
. 我建议使用
Jackson
。 If you use gradle you can add a dependency like this: compile("com.fasterxml.jackson.module:jackson-module-scala_2.12")
. 如果使用gradle,则可以添加如下依赖项:
compile("com.fasterxml.jackson.module:jackson-module-scala_2.12")
。 (Use appropriate scala version) (使用适当的scala版本)
Then, you can simply convert your data object to JSON like this: 然后,您可以像这样将数据对象简单地转换为JSON:
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
val json = valueToTree(messageData)
I'd strongly recommend you put your effort in Jackson, you'll need it a lot if you work with JSON. 我强烈建议您向Jackson投入精力,如果您使用JSON,将非常需要它。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.