[Spring Cloud][PubSub] Read avro messages from pubsub

Question

I have a Dataflow job which writes avro messages into PubSub:

PubsubIO.writeAvros(Session.class).to(sessionTopic)

Then, I want to process the messages in Spring application. In general, I'm using org.springframework.cloud.spring-cloud-gcp-starter-pubsub and org.springframework.cloud.spring-cloud-stream-schema dependencies.

@Bean
public PubSubInboundChannelAdapter messageChannelAdapter(
    @Qualifier("sessionInputChannel") MessageChannel inputChannel,
    PubSubOperations pubSubTemplate) {
  PubSubInboundChannelAdapter adapter = new PubSubInboundChannelAdapter(pubSubTemplate,
      "sessions-subscription");
  adapter.setOutputChannel(inputChannel);
  adapter.setMessageConverter(sessionMessageConverter());
  adapter.setAckMode(AckMode.MANUAL);
  return adapter;
}

private MessageConverter sessionMessageConverter() {
  AvroSchemaMessageConverter converter = new AvroSchemaMessageConverter();
  converter.setSchema(Session.getClassSchema());
  return converter;
}

@Bean
public MessageChannel sessionInputChannel() {
  return new DirectChannel();
}

@Bean
@ServiceActivator(inputChannel = "sessionInputChannel")
public MessageHandler messageReceiver() {
  return message -> {
    log.info("Message arrived! Payload: " + message.getPayload());
    Session session = (Session) message.getPayload();
    log.info("Session: " + session);
    AckReplyConsumer consumer = (AckReplyConsumer) message.getHeaders()
        .get(GcpPubSubHeaders.ACKNOWLEDGEMENT);
    consumer.ack();
  };
}

Unfortunately, I receive the following exception:

java.lang.NullPointerException: in com.example.model.Session null of com.example.model.Session
  at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:161) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62) ~[avro-1.8.2.jar:1.8.2]
  at org.springframework.cloud.stream.schema.avro.AbstractAvroMessageConverter.convertToInternal(AbstractAvroMessageConverter.java:183) ~[spring-cloud-stream-schema-2.0.0.RELEASE.jar:2.0.0.RELEASE]
  at org.springframework.messaging.converter.AbstractMessageConverter.toMessage(AbstractMessageConverter.java:201) ~[spring-messaging-5.0.6.RELEASE.jar:5.0.6.RELEASE]
  at org.springframework.messaging.converter.AbstractMessageConverter.toMessage(AbstractMessageConverter.java:191) ~[spring-messaging-5.0.6.RELEASE.jar:5.0.6.RELEASE]
  at org.springframework.cloud.gcp.pubsub.integration.inbound.PubSubInboundChannelAdapter.receiveMessage(PubSubInboundChannelAdapter.java:88) ~[spring-cloud-gcp-pubsub-1.0.0.M3.jar:1.0.0.M3]
  at com.google.cloud.pubsub.v1.MessageDispatcher$4.run(MessageDispatcher.java:405) ~[google-cloud-pubsub-0.43.0-beta.jar:0.43.0-beta]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_171]
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_171]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.NullPointerException: null
  at org.apache.avro.reflect.ReflectData.getField(ReflectData.java:158) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:164) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:90) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:191) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75) ~[avro-1.8.2.jar:1.8.2]
  at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:159) ~[avro-1.8.2.jar:1.8.2]

Can anyone tell me how to consume the avro messages? What I am doing wrong?

Answer 1

It looks like you're using the M3 release of the Spring GCP PubSub integration, and the line causing this error is in this file : https://github.com/spring-cloud/spring-cloud-gcp/blob/eeed548ec3a8a82bfd158a6f40d52a94b26ea61e/spring-cloud-gcp-pubsub/src/main/java/org/springframework/cloud/gcp/pubsub/integration/inbound/PubSubInboundChannelAdapter.java

On line 88, it seems that the payload is being retrieved as a UTF 8 string, which could be making the object no longer able to be converted back into the Session type.