[英]creating a cache of byte arrays
In my code, which often runs on servers I do not control the configuration of, I have collection of users and each user has a byte[]
array. 在我的代码中,它通常在服务器上运行,我不控制配置,我有用户集合,每个用户都有一个byte[]
数组。
Sometimes these byte[]
arrays are unique to the user. 有时这些byte[]
数组对用户来说是唯一的。 Often, though, there will be large numbers users with the exact same byte[]
array. 但是,通常会有大量用户使用完全相同的byte[]
数组。
I am trying to reduce the RAM consumption of my server. 我试图减少我的服务器的RAM消耗。
I've tried turning my byte[]
arrays into strings and interning them, but then I often run into PERM-GEN out-of-memory errors. 我已经尝试将我的byte[]
数组转换为字符串并实习它们,但后来我经常遇到PERM-GEN内存不足错误。 I also see a significant performance degradation with the encoding/decoding when I want to access the byte[]
array for a user, and I see much increased worse-case memory usage - presuambly strings are much larger than arrays. 当我想为用户访问byte[]
数组时,我也发现编码/解码会导致显着的性能下降,而且我看到更糟糕的内存使用情况 - 预备字符串比数组大得多。
How can I have a Set<SoftReference<byte[]>>
lookup when Java arrays are not hashable and SoftReferences do not wrap the hash of the object the point at either. 当Java数组不可清除时,如何使用Set<SoftReference<byte[]>>
查找,而SoftReferences不会将对象的哈希值包装在任何一个点上。 A Map<byte[],SoftReference<byte[]>>
is obviously also defeating itself because the key is itself and prevents collection; Map<byte[],SoftReference<byte[]>>
显然也会使自己失败,因为密钥本身并阻止了收集; and Set
is internally implemented in terms of Map
anyway. 并Set
来讲是内部实现Map
反正。
So how can I intern byte[]
arrays? 那我怎么能实习 byte[]
数组呢?
If you effectively have many identical arrays, use an HashSet<ByteBuffer>
as a cache. 如果您有效地拥有许多相同的数组,请使用HashSet<ByteBuffer>
作为缓存。 You can get the ByteBuffer array with method array()
and the ByteBuffer class has hashCode
and equals
methods. 您可以使用方法array()
获取ByteBuffer数组,并且ByteBuffer类具有hashCode
和equals
方法。 Of course it is better if your arrays are immutable. 当然,如果你的数组是不可变的,那就更好了。
EDIT2 The comment from @Will is exact, to be able to get back the array, use a WeakHashMap<ByteBuffer,WeakReference<ByteBuffer>>
and do something like that : EDIT2来自@Will的评论是准确的,为了能够取回数组,使用WeakHashMap<ByteBuffer,WeakReference<ByteBuffer>>
并执行类似的操作:
public byte[] internalize(byte[] bytes) {
ByteBuffer wrapped = ByteBuffer.wrap(bytes);
if(cache.containsKey(wrapped)) {
wrapped = cache.get(wrapped).get();
}
else {
cache.put(wrapped, new WeakReference<ByteBuffer>(wrapped);
}
return wrapped.array();
}
I've tried turning my byte[] arrays into strings and interning them, but then I often run into PERM-GEN out-of-memory errors. 我已经尝试将我的byte []数组转换为字符串并实习它们,但后来我经常遇到PERM-GEN内存不足错误。
I agree that you need something like String.intern()
, but the standard implementation is native
, so not much joy. 我同意你需要像String.intern()
这样的东西,但是标准的实现是native
,所以没有多少快乐。
You could have a Map<Integer,Collection<SoftReference<byte[]>>>
, using the hash code of the byte arrays as the Map
key. 你可以使用Map<Integer,Collection<SoftReference<byte[]>>>
,使用字节数组的哈希码作为Map
键。 Your intern
method could then look up the set of existing byte arrays with the same has code as the given byte arrays. 然后,您的intern
方法可以使用与给定字节数组相同的代码查找现有字节数组。 With a good hash code that should give a small set of arrays to examine for a match. 使用一个好的哈希代码,它应该提供一小组数组来检查匹配。
Edit: To clarify: 编辑:澄清:
Something like this: 像这样的东西:
class ByteArrayCache
{
private final Map<Integer,Collection<SoftReference<byte[]>> map = new ...;
public final byte[] intern(byte[] byteArray)
{
final int hash = Arrays.hashCode(byteArray);
final Collection<SoftReference<byte[]>> arrays = map.get(hash);
if (arrays != null) {
// Search through arrays for a match, and return the match.
// If no match found, add byteArray to the collection and return it
} else {
// create a new map entry, add byteArray to it, and return byte array
}
}
}
I would implement a cache based on Guava weak values map. 我会基于Guava弱值映射实现缓存。 It guarantees that if theres no more strong references to byte array the entry will be automatically removed. 它保证如果没有对字节数组的更强引用,则会自动删除该条目。
class Cache {
private final ConcurrentMap<Key, byte[]> map = new MapMaker().weakValues().makeMap();
private static class Key {
byte[] a;
int hash;
Key(byte[] a) {
this.a = a;
hash = Arrays.hashCode(a);
}
@Override
public int hashCode() {
return hash;
}
@Override
public boolean equals(Object obj) {
if (obj instanceof Key) {
return Arrays.equals(a, ((Key) obj).a);
}
return false;
}
}
public byte[] intern(byte[] a) {
byte[] a1 = map.putIfAbsent(new Key(a), a);
if (a1 != null) {
return a1;
}
return a;
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.