How to properly communicate to a Kafka Broker through a socket?

Question

Recently I have been trying to create a client library for Kafka in JavaScript (similar to KafkaJS).

I looked into how KafkaJS does it and I successfully fetched a messaged from a topic by opening a socket and writing the Kafka protocol.

The issue I am facing is, right after I send the first request (Fetch) to the broker and receive an answer, the broker sends a FIN packet and ends the socket.

I think this behavior is not normal and I searched in the official documents and also in KafkaJS code, I couldn't find anything.

const socket = new net.Socket();
socket.connect(9092, 'localhost');
socket.setKeepAlive(true, 60000);

socket.write(payload.buffer, 'binary');

//rest of the events (end, data, error) are here.

Answer 1

Kafka has its own TCP protocol, which is implemented in several Nodejs libraries. Would be best to use those rather than implementing it yourself, but if you need to, the first action is to send a bootstrap request, not the payload, then producer records are sent in batches, to specific topic partitions of certain brokers (even if there's only one broker).

Not sure about your FIN issue, but if that's what the server returns, then somehow other clients would be handling it

Also, KafkaJS is a server side library. Ideally, you wouldn't have a client (browser) communicating directly to Kafka