简体   繁体   English

kafka 流状态存储保存在哪里?

[英]Where are kafka streams state stores saved?

I have a kafka streams application in which I am using state stores.我有一个 kafka 流应用程序,我在其中使用状态存储。 I have read from a few places that state stores are backed in local rocksdb instance under /tmp directory.我从几个地方读到状态存储支持在 /tmp 目录下的本地rocksdb实例中。 But in my application if I stop it and remove the application kafka stream directory from /tmp, I still am able to get my state store restored to previous state.但是在我的应用程序中,如果我停止它并从 /tmp 中删除应用程序 kafka 流目录,我仍然能够将我的状态存储恢复到以前的状态。 Which means either state is not stored in rocksdb at all or it is stored in both kakfa and local rocksdb.这意味着状态根本不存储在rocksdb中,或者同时存储在kakfa和本地rocksdb中。 So my question is, where is the state stored and since I am using docker, is there any performance or any other benefit of persisting /tmp/kafka-streams directory to a volume?所以我的问题是,状态存储在哪里,并且由于我使用的是 docker,将 /tmp/kafka-streams 目录持久化到卷是否有任何性能或任何其他好处? My test code is below我的测试代码如下

StoreBuilder<KeyValueStore<String, String>> testStoreBuilder = Stores
            .keyValueStoreBuilder(Stores.persistentKeyValueStore("test-store"), Serdes.String(), Serdes.String())
            .withCachingEnabled();
    builder.addStateStore(testStoreBuilder);

    final KStream<String, String> sensorDataStream = builder.stream("test");

    sensorDataStream.transformValues(() -> new ValueTransformer<String, String>() {
        private KeyValueStore<String, String> testStore;

        @SuppressWarnings("unchecked")

        @Override
        public void init(ProcessorContext context) {
            testStore = (KeyValueStore<String, String>) context.getStateStore("test-store");
        }

        @Override
        public String transform(String value) {
            LocalDateTime start = LocalDateTime.now();
            System.out.println(testStore.get("test"));
            testStore.put("test", value);
            System.out.println("DURATION:: " + Duration.between(start, LocalDateTime.now()).toMillis());
            return null;
        }

        @Override
        public void close() {
        }
    }, "test-store");

When using Apache Kafka dependencies, then Streams are stored in /tmp .当使用 Apache Kafka 依赖项时,Streams 存储在/tmp If using the Confluent dependencies, it's under /var/lib , I think.如果使用 Confluent 依赖项,它在/var/lib ,我想。

RocksDB is backed by a consumed changelog topic that is stored in Kafka. RocksDB 由存储在 Kafka 中的消费变更日志主题支持。

If you want clear all the state, use the application reset tool如果要清除所有状态,请使用应用程序重置工具

since I am using docker因为我正在使用 docker

That shouldn't matter unless you remove/rebuild the image between application runs since Docker still writes to your host's filesystem除非您在应用程序运行之间删除/重建映像,否则这无关紧要,因为 Docker 仍会写入主机的文件系统

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM