How to save kafka messages into a file using asp.net

Question

Hi I have created a console application for kafka consumer for receiving messages.

class Program
{
    static void Main(string[] args)
    {
        string topic = "IDGTestTopic";
        Uri uri = new Uri("http://localhost:9092");
        var options = new KafkaOptions(uri);
        var router = new BrokerRouter(options);
        var consumer = new Consumer(new ConsumerOptions(topic, router));
        foreach (var message in consumer.Consume())
        {
            Console.WriteLine(Encoding.UTF8.GetString(message.Value));
            //Saving messages in files
            string lines = Encoding.UTF8.GetString(message.Value);
            System.IO.File.WriteAllText(@"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", lines);
        }

    }
}

But it only store the current messages. if you see the console all the messages are displaying.

But if you see the text file it only contain current messages

How to save all the messages in a file ?

Answer 1

For each message, System.IO.File.WriteAllText overwrites the file and therefore the created file will contain only the latest message.

In order to keep all the messages in a single file, you can replace System.IO.File.WriteAllText with System.IO.File.AppendAllText as shown below:

foreach (var message in consumer.Consume()) {
    Console.WriteLine(Encoding.UTF8.GetString(message.Value));
    //Saving messages in files
    string lines = Encoding.UTF8.GetString(message.Value);
    System.IO.File.AppendAllText(@"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", lines);
}

According to the docs,

File.AppendAllText Method (String, String)

Opens a file, appends the specified string to the file, and then closes the file. If the file does not exist, this method creates a file, writes the specified string to the file, then closes the file.

and File.WriteAllText Method (String, String)

Creates a new file, writes the specified string to the file, and then closes the file. If the target file already exists, it is overwritten.

Answer 2

Every time you consume a message, you overwrite the whole file:

System.IO.File.WriteAllText

You need to do this outside the consume loop.

Answer 3

@ Giorgos Myrianthous Your previous second choice was better in some ways. Appending to a stringbuilder and writing only once outside the loop to a file is most likely much faster then going through IO many times in every loop. Here is what I was suggesting:

StringBuilder linebuilder = new StringBuilder();  //this line outside the loop

foreach (var message in consumer.Consume()) {
    Console.WriteLine(Encoding.UTF8.GetString(message.Value));
    //Saving messages in files
    linebuilder.Append(Encoding.UTF8.GetString(message.Value)); //this line inside the loop
}

System.IO.File.AppendAllText(@"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", linebuilder.ToString(());