如何让php api以二进制格式登录kafka

Question

Let's assume we have an PHP api which runs under apache webserver. I'd like to ask how it would be possible to put logs from PHP to kafka in a structured format (let's say avro or protobuf because it would transfer specific business contracts, not just a pure apache logs). Php is not in my sphere or expertise so it might be dumb, but the solution could be just to send messages to kafka from php itself. But given that it runs under webserver most likely it would have to create connections to kafka for every call. Another possible solution would be to log to file and then use something like filebeat to put logs to kafka. But I'm not sure how it would work if we used, as I've already mentioned, structured format. I doubt filebeat would be able to read that binary log file. Please, let me know if it can. So my question is - what is the best way to get php specific logs into kafka in a rigid and scalable manner? Thanks in advance.

Answer 1

Filebeat would not read binary logs. It would be used to parse your plaintext messages into some more structured/binary format before sending to Kafka (I'm not certain Filebeat's Kafka serialization is pluggable, though)

On the other hand, yes, there's a php-librdkafka library that you can try, but as you said - it'd require some persistent connection to Kafka, compared to having a type of "two-commit" system with files in that every line of the file tracks progress and can withstand process restarts due to Kafka connection errors, for example

Fluentbit or rsyslog or (ideally if you are truly log processing) Splunk forwarder also work...