[英]Where are kafka streams state stores saved?
I have a kafka streams application in which I am using state stores.我有一个 kafka 流应用程序,我在其中使用状态存储。 I have read from a few places that state stores are backed in local rocksdb instance under /tmp directory.
我从几个地方读到状态存储支持在 /tmp 目录下的本地rocksdb实例中。 But in my application if I stop it and remove the application kafka stream directory from /tmp, I still am able to get my state store restored to previous state.
但是在我的应用程序中,如果我停止它并从 /tmp 中删除应用程序 kafka 流目录,我仍然能够将我的状态存储恢复到以前的状态。 Which means either state is not stored in rocksdb at all or it is stored in both kakfa and local rocksdb.
这意味着状态根本不存储在rocksdb中,或者同时存储在kakfa和本地rocksdb中。 So my question is, where is the state stored and since I am using docker, is there any performance or any other benefit of persisting /tmp/kafka-streams directory to a volume?
所以我的问题是,状态存储在哪里,并且由于我使用的是 docker,将 /tmp/kafka-streams 目录持久化到卷是否有任何性能或任何其他好处? My test code is below
我的测试代码如下
StoreBuilder<KeyValueStore<String, String>> testStoreBuilder = Stores
.keyValueStoreBuilder(Stores.persistentKeyValueStore("test-store"), Serdes.String(), Serdes.String())
.withCachingEnabled();
builder.addStateStore(testStoreBuilder);
final KStream<String, String> sensorDataStream = builder.stream("test");
sensorDataStream.transformValues(() -> new ValueTransformer<String, String>() {
private KeyValueStore<String, String> testStore;
@SuppressWarnings("unchecked")
@Override
public void init(ProcessorContext context) {
testStore = (KeyValueStore<String, String>) context.getStateStore("test-store");
}
@Override
public String transform(String value) {
LocalDateTime start = LocalDateTime.now();
System.out.println(testStore.get("test"));
testStore.put("test", value);
System.out.println("DURATION:: " + Duration.between(start, LocalDateTime.now()).toMillis());
return null;
}
@Override
public void close() {
}
}, "test-store");
When using Apache Kafka dependencies, then Streams are stored in /tmp
.当使用 Apache Kafka 依赖项时,Streams 存储在
/tmp
。 If using the Confluent dependencies, it's under /var/lib
, I think.如果使用 Confluent 依赖项,它在
/var/lib
,我想。
RocksDB is backed by a consumed changelog topic that is stored in Kafka. RocksDB 由存储在 Kafka 中的消费变更日志主题支持。
If you want clear all the state, use the application reset tool如果要清除所有状态,请使用应用程序重置工具
since I am using docker
因为我正在使用 docker
That shouldn't matter unless you remove/rebuild the image between application runs since Docker still writes to your host's filesystem除非您在应用程序运行之间删除/重建映像,否则这无关紧要,因为 Docker 仍会写入主机的文件系统
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.