Found directory not in the form of topic-partition . Kafka's log directories (and children) should only contain Kafka topic data

Question

[2021-04-05 07:51:32,180] ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.KafkaException: Found directory /var/lib/kafka/data, 'data' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data. (kafka.log.LogManager)
[2021-04-05 07:51:32,189] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Found directory /var/lib/kafka/data, 'data' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data.
        at kafka.log.Log$.exception$1(Log.scala:2680)
        at kafka.log.Log$.parseTopicPartitionName(Log.scala:2685)
        at kafka.log.LogManager.loadLog(LogManager.scala:258)
        at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:352)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
[2021-04-05 07:51:32,195] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)

why am i getting this error i couldn't solve it please help?

Answer 1

Means you've set log.dirs to include /var/lib/kafka , but there's a data subdirectory there that doesn't match the format [topic]-[partition] . Depending on what's in the data folder, fix might include using that, or making a new one that's empty

In a production Kafka installation, you'd mount individual disks for storing Kafka data rather than store it on the OS drive under /var