Logstash pipeline issues when sending to multiple Kafka topics

Question

I am using Logstash to extract change data from an SQL Server DB and send it to different Kafka topics. Some Logstash config files send to the Ticket Topic others to the Availability topic If I run just the configs that send to the Ticket topic on their own using the pipeline it works fine. If I run the configs for availability topic on their own in a pipeline they send the data ok.

However when I include the configs to send to both topics together I get the error. Please see extract from the logs. This time the availability topic failed other times the ticket topic fails.

[2021-03-22T07:30:00,172][WARN ][org.apache.kafka.clients.NetworkClient][AvaililityDOWN] [Producer clientId=Avail_down1] Error while fetching metadata with correlation id 467 : {dcsvisionavailability=TOPIC_AUTHORIZATION_FAILED}
[2021-03-22T07:30:00,172][ERROR][org.apache.kafka.clients.Metadata][AvaililityDOWN] [Producer clientId=Avail_down1] Topic authorization failed for topics [dcsvisionavailability]
[2021-03-22T07:30:00,203][INFO ][logstash.inputs.jdbc     ][Ticket1][a296a0df2f603fe98d8c108e860be4d7a17f840f9215bb90c5254647bb9c37cd] (0.004255s) SELECT sys.fn_cdc_map_lsn_to_time(__$start_lsn) transaction_date, abs(convert(bigint, __$seqval)) seqval, * FROM cdc.dbo_TICKET_CT where ( __$operation = 2 or __$operation = 4) and modified_date > '2021-03-22T07:27:00.169' order by modified_date ASC
[2021-03-22T07:30:00,203][INFO ][logstash.inputs.jdbc     ][AvailabilityMAXUP][7805e7bd44f20b373e99845b687dc15d7c2a3de084fb4424dd492be93b39b64a] (0.004711s) With Logstash as(
SELECT sys.fn_cdc_map_lsn_to_time(__$start_lsn) transaction_date, abs(convert(bigint, __$seqval)) seqval, *
FROM cdc.dbo_A_TERM_MAX_UPTIME_DAY_CT
)
select * from Logstash
where ( __$operation = 2 or __$operation = 4 or __$operation = 1 ) and TMZONE = 'Etc/UTC' and transaction_date > '2021-03-22T07:15:00.157' order by seqval ASC
[2021-03-22T07:30:00,281][WARN ][org.apache.kafka.clients.NetworkClient][AvailabilityMAXUP] [Producer clientId=Avail_MaxUp1] Error while fetching metadata with correlation id 633 : {dcsvisionavailability=TOPIC_AUTHORIZATION_FAILED}
[2021-03-22T07:30:00,281][ERROR][org.apache.kafka.clients.Metadata][AvailabilityMAXUP] [Producer clientId=Avail_MaxUp1] Topic authorization failed for topics [dcsvisionavailability]
[2021-03-22T07:30:00,297][WARN ][org.apache.kafka.clients.NetworkClient][AvaililityDOWN] [Producer clientId=Avail_down1] Error while fetching metadata with correlation id 468 : {dcsvisionavailability=TOPIC_AUTHORIZATION_FAILED}
[2021-03-22T07:30:00,297][ERROR][org.apache.kafka.clients.Metadata][AvaililityDOWN] [Producer clientId=Avail_down1] Topic authorization failed for topics [dcsvisionavailability]
[2021-03-22T07:30:00,406][WARN ][org.apache.kafka.clients.NetworkClient][AvailabilityMAXUP] [Producer clientId=Avail_MaxUp1] Error while fetching metadata with correlation id 634 : {dcsvisionavailability=TOPIC_AUTHORIZATION_FAILED}
[2021-03-22T07:30:00,406][ERROR][org.apache.kafka.clients.Metadata][AvailabilityMAXUP] [Producer clientId=Avail_MaxUp1] Topic authorization failed for topics [dcsvisionavailability]
[2021-03-22T07:30:00,406][WARN ][logstash.outputs.kafka   ][AvailabilityMAXUP][3685b3e90091e526485060db8df552a756f11f0f7fd344a5051e08b484a8ff8a] producer send failed, dropping record {:exception=>Java::OrgApacheKafkaCommonErrors::TopicAuthorizationException, :message=>"Not authorized to access topics: [dcsvisionavailability]", :record_value=>"<A_TERM_MAX_UPTIME_DAY>\

This is the output section of the availability config

output {
    kafka {
      bootstrap_servers =>  "namespaceurl.windows.net:9093"
      topic_id => "dcsvisionavailability"
      security_protocol => "SASL_SSL"
      sasl_mechanism => "PLAIN"
      jaas_path => "C:\Logstash\keys\kafka_sasl_jaasAVAILABILITY.java"
      client_id => "Avail_MaxUp1"
      codec  => line {
      format => "<A_TERM_MAX_UPTIME_DAY>
    <stuff deleted>"
      }
}

}

The pipeline.yml file has this in it

## Ticket Topic
- pipeline.id: Ticket1
  path.config: "TicketCT2KafkaEH8.conf"
  queue.type: persisted
  
- pipeline.id: PublicComments1
  path.config: "Public_DiaryCT2KafkaEH1.conf"
  queue.type: persisted

##  - Availability topic 
- pipeline.id: AvailabilityDOWN
  path.config: "Availability_Down_TimeCT2KafkaEH3.conf"
  queue.type: persisted
  
- pipeline.id: AvailabilityMAXUP
  path.config: "Availability_Max_UptimeCT2KafkaEH2.conf"
  queue.type: persisted   

I have tried running in different instances and yes that works where I have the Pipeline running and open another command window and run one other config sending to a different topic (for this I specify a different --path.data )

However with 40 configs going to 4 different topics I don't really want to run so lots of instances in parallel. Any advice welcomed

Answer 1

I have been able to resolve the issue. It was to do with the jaas_path file.

I had a different jaas_path file for each topic that specified the topic in the string as follows in the EntityPath.

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="$ConnectionString"
password="Endpoint=sb://<stuff redacted>.windows.net/;SharedAccessKeyName=keyname;SharedAccessKey=<key redacted>;EntityPath=dcsvisionavailability";
};

When I provided common key from Event Hub for use with all the topics that does not have the;EntityPath=topicname at the end it worked.

This makes sense since I had already specified the topic in the line topic_id => "dcsvisionavailability" in the logstash conf file.

Answer 2

I'm glad it worked Walter.

For others who may look up for solution on this issue, the issue here was same shareAccessKeyName was used with different tokens for different Kafka topics for authorization at EventHub.

That is why, it all worked fine when requests targeted to only one topic came in for authorization.

When requests for different topics came to EventHub at the same time with same shareAccessKeyName but with different tokens, only one would go through but the other would exception out due to conflict of token.

Different options for solving this were

1. use same token for all requests (for all topics)
2. use different shareAccessKeyName with different token for each topic.

Walter chose#1 and needed topic name to be removed in password (as EventHub might have had topic wise authorization).

For topic wise authorization, solution #2 will be better.