How to capture Kafka SSL related failures into Serilog sinks C# console app

Question

I am trying to produce a topic through Kafka in C# .NET Core and receiving SSL related failures, because of bad certificate I am using.

[15:19:49 INF] Log in Program.cs
%3|1620890390.755|FAIL|samplepublisher#producer-1| [thrd:ssl://bootstrap.data.test.co:9094/bootstrap]: ssl://bootstrap.data.test.co:9094/bootstrap: SSL handshake failed: error:140900:SSL routines:ssl3_get_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (add broker's CA certificate to the Windows Root certificate store) (after 18ms in state SSL_HANDSHAKE)

This issue was fixed, and it was something related with certificates and all. But the real problem was, this FAIL message from Kafka was not logged into anywhere in our .NET Core 3.1 application. We are using SeriLog and below is the code we used.

private static void ConfigureServices(IServiceCollection services, IConfiguration configuration)
        {
            services.AddLogging(configure => configure.AddSerilog());

            services.Configure<LoggerFilterOptions>(options => options.MinLevel = LogLevel.Trace);
        }

This is the main program code.

Log.Logger = new LoggerConfiguration()
    .WriteTo.File("logs\\log.txt", rollingInterval: RollingInterval.Day, shared: true)
    .WriteTo.Console()
    .WriteTo.Debug()
    .CreateLogger();

var serviceCollection = new ServiceCollection();
ConfigureServices(serviceCollection, configuration);

var serviceProvider = serviceCollection.BuildServiceProvider();

var logger = serviceProvider.GetService<ILogger<Program>>();

logger.LogInformation("Log in Program.cs");

try
    {
       var config = new ProducerConfig
       {
         BootstrapServers = Server,
         SslCaLocation = "data.cer",
         SecurityProtocol = SecurityProtocol.Ssl,
       };

 using var producer = new ProducerBuilder<Null, string>(config).Build();
       await producer.ProduceAsync(Topic, new Message<Null, string> { Value = "test message from"});
            
        }
        catch (Exception e)
        {
            Console.WriteLine($"publish failed in {sw.ElapsedMilliseconds} seconds", e);
            throw;
        }

I think the issue is that those error are coming from the depths of the confluence library and not from our code. I am looking for a way if I can get those logs also into our SeriLog sinks.

Answer 1

I believe it's ErrorHandler you are looking for. Pass an instance of the Serilog logger into it, eg:

using var producer = new ProducerBuilder<Null, string>(config)
    .SetErrorHandler((_, error) => logger.LogWarning("Error in Kafka Handler: {}", error))
    .Build();