无法使用 Apache Beam Java sdk 连接到 PulsarIO

Question

while executing below code to connect to apache pulsar using apache beam PulsarIO in java sdk. Getting below error while adding pulsar client in beam pipeline.

Beam version 2.40, 2.41 javaSE 1.8

import java.io.*;
import org.apache.beam.sdk.*;
import org.apache.beam.io.pulsar.*;
import org.apache.beam.sdk.options.*;
import org.apache.beam.sdk.transforms.*;
import org.apache.beam.sdk.values.*;
import org.apache.beam.transforms.SerializableFunction;
import org.apache.pulsar.client.api.*;
import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.apache.pulsar.client.impl.auth.AuthenticationTls;
import java.util.*;

public class BeamPulsar{
 public static void main(String args[]) throws PulsarClientException{
 PipelineOptions options=PipelineOptionsFactory.as(PipelineOptions.class);
 Pipeline p= Pipeline.create(options=options);
 PulsarClient pulsarClient;

 String TLSCERTFILE='filepath'
 String TLSKEYFILE='filepath'
 String CACERTFILE='filepath'
 String BROKER_URL='pulsar+ssl://'
 String TOPIC_NAME='topic'
 Map<String, String> authParams=new HashMap<>();
 authParams.put("tlsCertFile",TLSCERTFILE);
 authParams.put("tlsKeytFile",TLSKEYFILE);
 Authentication tlsAuth =AuthenticationFactory.create(AuthenticationTls.class.getName(),authParams);
 pulsarClient=PulsarClient.builder().serviceUrl(BROKER_URL).tlsTrustCertsFilePath(CACERTFILE).authentication(tlsAuth).build()
PCollection<PulsarMessage> records=p.apply("read from pulsar", PulsarIO.read()
                                    .withTopic(TOPIC_NAME)
                                    .withPulsarClient((SerializableFunction<String, PulsarClient>)pulsarClient)
                                    .withPublishTime()
                                    .withClientUrl(BROKER_URL)
                                    .withAdminUrl(BROKER_URL));
p.run().waitUntilFinish();
}}

Error: Exception in thread "main" java.lang.ClassException: class org.apache.pulsar.client.impl.PulsarClientImpl cannot be cast to class org.apache.beam.sdk.transform.SerializableFunction(org.apache.pulsar.client.impl.PulsarClientImpl and org.apache.beam.sdk.transforms.SerializableFunction are in unnamed module of loader app)

Answer 1

You will want to put the construction of the PulsarClient inside of an implementation of SerializableFunction and not in your main. The client needs to be constructed in the remote process actually running the IO code, not the machine that submits the pipeline.