如何使用 Scala 中的 Beam 將 Kafka 消費到 MySQL

Question

我想使用來自 Kafka ZF7B44CFFAFD5C52223D5498196C8A2E7BZ 的數據並使用 Apache Beam 將其寫入 MySQL。 顯示錯誤Cannot resolve overloaded method 'apply' .apply(JdbcIO.write[KV[Integer, String]]()...

這是我的完整代碼：

def main(cmdlineArgs: Array[String]): Unit = {
    val options = PipelineOptionsFactory.create()

    val pipeline = Pipeline.create(options)

    pipeline
        .apply(KafkaIO.read[Long, String]()
          .withBootstrapServers("kafka-stag.c.kata-production.internal:31090")
          .withTopic("topic-saya")
          .withKeyDeserializer(classOf[LongDeserializer])
          .withValueDeserializer(classOf[StringDeserializer])
          .updateConsumerProperties(
            ImmutableMap.of("auto.offset.reset", "earliest".asInstanceOf[Object])
          )
          .withMaxNumRecords(5)
          .withoutMetadata()
        )
          .apply(MapElements.into(TypeDescriptors.strings()))
          .apply(JdbcIO.write[KV[Integer, String]]()
            .withDataSourceConfiguration(DataSourceConfiguration.create(
              "com.mysql.jdbc.Driver",
              "jdbc:mysql://localhsot:3306/mydb")
            .withUsername("root")
            .withPassword("secret"))
            .withStatement("insert into Person values(?, ?)")
            .withPreparedStatementSetter(
                (element: KV[Integer, String], query: PreparedStatement) => {
                  query.setInt(1, element.getKey())
                  query.setString(2, element.getValue())
              })
          )


    pipeline.run().waitUntilFinish()
}

我應該怎么做才能解決這個問題？

Answer 1

我發現在調試 scala 代碼時分解長鏈方法很有用。

在這種情況下，您將三個apply(...)鏈接在一起：

// Heads up: you'll want to specify java.lang.Long when working
// with Java code and generics... this has bitten me *many* times.

val x: PCollection[KV[java.lang.Long, String]] = pipeline
    .apply(KafkaIO.read[java.lang.Long, String]()
        .withThisAndThatConfig(...))

// Probably not what you're looking for...
// MapElements requires via() to modify the element.

val y: PCollection[String] = x
    .apply(MapElements.into(TypeDescriptors.strings()))

// Error happens below, the JdbcIO.write is expecting to be applied
// to a PCollection[KV[Integer, String]], which is not what it's getting.

val z: PDone = y
    .apply(JdbcIO.write[KV[Integer, String]]()
            .withThisAndThatConfig(...))

如果我了解您要查找的內容，您可能需要仔細查看中間的apply並確保它為 JdbcIO 發出正確的元素。 處理這兩種語言有點麻煩，但你最終可能會得到類似的結果：

val keystoInt: SerializableFunction[KV[java.lang.Long, String], KV[Integer, String]] =
    (input: KV[java.lang.Long, String]) => KV.of(input.getKey.intValue, input.getValue)
val y: PCollection[KV[Integer, String]] = x
    .apply(MapElements
        .into(TypeDescriptors.kvs(TypeDescriptors.integers, TypeDescriptors.strings))
        .via(keysToInt))