简体   繁体   中英

Data sharing in Couchbase's memcached implementation between Java and .Net

I am trying to share data stored in a couchbase memcached bucket between Java and .Net.

I was able to read a string set in Java in .Net but whenever I try to read a string set in .Net in Java the result is Null.

So is it possible to exchange data between .Net and Java in memcache buckets in a couchbase server.

Thanks for the reply, I figured it out.

The reason .NET is able to read strings set in Java is because the enyimMemcached library interprets the cached item as a string if it does not recognize the flag.

So in order to be able to read the strings in Java, I simply created my own custom transcoder by extending the SpyObject and set it in away such as it ignores the flag. I then pass the custom transcoder with my get call like this,

_obj = GetMemcachedClient().get(key, new StringTranscoder())

My StringTranscoder class looks like this,

  /**
 * Copyright (C) 2006-2009 Dustin Sallings
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALING
 * IN THE SOFTWARE.
 */

package cachingjavatestapp;

import java.io.IOException;
import net.spy.memcached.CachedData;
import net.spy.memcached.compat.SpyObject;
import net.spy.memcached.transcoders.Transcoder;
import net.spy.memcached.transcoders.TranscoderUtils;

/**
 * Transcoder that serializes and unserializes longs.
 */
public final class StringTranscoder extends SpyObject implements
    Transcoder<String> {

  private static final int FLAGS = 0;

  public boolean asyncDecode(CachedData d) {
    return false;
  }

  public CachedData encode(java.lang.String l) {
        try{
            return new CachedData(FLAGS, l.getBytes("UTF-8"), getMaxSize());
        }
        catch (Exception e){
            return null;
        }
    }


  public String decode(CachedData d) {
        try{
            return new String(d.getData(), "UTF-8");
        }catch(Exception e){
            return null;
        }
  }

  public int getMaxSize() {
    return CachedData.MAX_SIZE;
  }
}

In order to be able to exchange data between .NET and Java. I simply used the json.net library and the gson library to serialize objects and pass the json strings to memcached where it gets picked up as a string then deserialized using the json libraries.

Regards,

Yes, it is possible. In the case of the Java client, it has a built in "transcoder" that will handle converting a java.lang.String to bytes with appropriate encoding (UTF-8 I think? I'd have to check). The .NET side would be able to read this data back in.

Where things get sticky is how each client library stores a string. In memcached protocol, the recommended but not required way to do this is with the flags. The challenge is, each client library does flags differently, something the Couchbase client library developers are looking to resolve into a common set of flags.

This means, for now, to normalize how the data is stored between the two client libraries, you may have to set up the client library a particular way. For example, Java has customizable transcoders and you can extend one of the existing transcoders to read/write your strings with the flags that the .NET client library is using.

Let me know what client libraries you're using, and I'll update this with an example.

I know this is a rather old question, but I had the same problem and thought I'd share my current solution. The following transcoder for EnyimMemcached more closely matches the way types are serialized/flagged in spymemcached. Obviously this is NOT going to work if you attempt to serialize objects between .Net and Java; but it will let you work with more than just strings.

https://github.com/mikeleedev/EnyimMemcached/blob/master/Enyim.Caching/Memcached/Transcoders/SpymemcachedTranscoder.cs

    public class SpymemcachedTranscoder : ITranscoder
    {
        #region Private Members
        // General flags
        private const uint SERIALIZED = 1; //00000000 00000001
        private const uint COMPRESSED = 2; //00000000 00000010 <-- TODO - add support for compression
        private const uint NOFLAG = 0; //00000000 00000000

        // Special flags for specially handled types.
        private const uint SPECIAL_MASK = 0xff00; //11111111 00000000
        private const uint SPECIAL_BOOLEAN = (1 << 8); //00000001 00000000
        private const uint SPECIAL_INT = (2 << 8); //00000010 00000000
        private const uint SPECIAL_LONG = (3 << 8); //00000011 00000000
        private const uint SPECIAL_DATE = (4 << 8); //00000100 00000000
        private const uint SPECIAL_BYTE = (5 << 8); //00000101 00000000
        private const uint SPECIAL_FLOAT = (6 << 8); //00000110 00000000
        private const uint SPECIAL_DOUBLE = (7 << 8); //00000111 00000000
        private const uint SPECIAL_BYTEARRAY = (8 << 8); //00001000 00000000

        private readonly ArraySegment<byte> NullArray = new ArraySegment<byte>(new byte[0]);
        private readonly DateTime EPOCH_START_DATETIME = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
        private readonly SpyMemcachedTranscoderUtils _spyTranscoderUtil = new SpyMemcachedTranscoderUtils(true); 
        #endregion

        #region Serialize/Deserialize
        CacheItem ITranscoder.Serialize(object value)
        {
            return this.Serialize(value);
        }

        object ITranscoder.Deserialize(CacheItem item)
        {
            return this.Deserialize(item);
        }

        protected virtual CacheItem Serialize(object value)
        {
            // raw data is a special case when some1 passes in a buffer (byte[] or ArraySegment<byte>)
            if (value is ArraySegment<byte>)
            {
                // ArraySegment<byte> is only passed in when a part of buffer is being 
                // serialized, usually from a MemoryStream (To avoid duplicating arrays 
                // the byte[] returned by MemoryStream.GetBuffer is placed into an ArraySegment.)
                return new CacheItem(SPECIAL_BYTEARRAY, (ArraySegment<byte>)value);
            }

            var tmpByteArray = value as byte[];

            // - or we just received a byte[]. No further processing is needed.
            if (tmpByteArray != null)
            {
                return new CacheItem(SPECIAL_BYTEARRAY, new ArraySegment<byte>(tmpByteArray));
            }

            uint flags = NOFLAG;
            ArraySegment<byte> data;
            TypeCode code = value == null ? TypeCode.Empty : Type.GetTypeCode(value.GetType());

            switch (code)
            {
                case TypeCode.Empty:
                case TypeCode.DBNull:
                    flags = SPECIAL_BYTEARRAY;
                    data = this.SerializeNull();
                    break;
                case TypeCode.String:
                    flags = NOFLAG;
                    data = this.SerializeString((String)value);
                    break;
                case TypeCode.Int64:
                    flags = SPECIAL_LONG;
                    data = this.SerializeInt64((Int64)value);
                    break;
                case TypeCode.Int32:
                    flags = SPECIAL_INT;
                    data = this.SerializeInt32((Int32)value);
                    break;
                case TypeCode.Boolean:
                    flags = SPECIAL_BOOLEAN;
                    data = this.SerializeBoolean((Boolean)value);
                    break;
                case TypeCode.DateTime:
                    flags = SPECIAL_DATE;
                    data = this.SerializeDateTime((DateTime)value);
                    break;
                case TypeCode.Byte:
                    flags = SPECIAL_BYTE;
                    data = this.SerializeByte((byte)value);
                    break;
                case TypeCode.Single: //float
                    flags = SPECIAL_FLOAT;
                    data = this.SerializeSingle((float)value);
                    break;
                case TypeCode.Double:
                    flags = SPECIAL_DOUBLE;
                    data = this.SerializeDouble((double)value);
                    break;
                default:
                    flags = SERIALIZED;
                    data = this.SerializeObject(value);
                    break;
            }

            //TODO - determine when to apply compression and do it

            return new CacheItem(flags, data);
        }

        protected virtual object Deserialize(CacheItem item)
        {
            if (item.Data.Array == null)
                return null;

            byte[] data = new byte[item.Data.Count];
            Array.Copy(item.Data.Array, item.Data.Offset, data, 0, item.Data.Count);

            //TODO - compression support
            //if ((item.Flags & COMPRESSED) != 0)
            //{
            //    data = Decompress(item.Data);
            //}

            if ((item.Flags & SERIALIZED) != 0)
            {
                return DeserializeObject(data);
            }

            uint flags = item.Flags & SPECIAL_MASK;
            if (flags == NOFLAG)
            {
                return DeserializeString(data);
            }
            else
            {
                switch (flags)
                {
                    case SPECIAL_BYTEARRAY:
                        return data;
                    case SPECIAL_BOOLEAN:
                        return this.DeserializeBoolean(data);
                    case SPECIAL_INT:
                        return this.DeserializeInt32(data);
                    case SPECIAL_LONG:
                        return this.DeserializeInt64(data);
                    case SPECIAL_DATE:
                        return this.DeserializeDateTime(data);
                    case SPECIAL_BYTE:
                        return this.DeserializeByte(data);
                    case SPECIAL_FLOAT:
                        return this.DeserializeSingle(data);
                    case SPECIAL_DOUBLE:
                        return this.DeserializeDouble(data);
                    default:
                        throw new InvalidOperationException(string.Format("SpyTranscoder undecodable with flags: {0}", flags));
                }
            }
        }     
        #endregion

        #region Typed Serialization

        protected virtual ArraySegment<byte> SerializeNull()
        {
            return NullArray;
        }

        protected virtual ArraySegment<byte> SerializeString(string value)
        {
            return new ArraySegment<byte>(Encoding.UTF8.GetBytes((string)value));
        }

        protected virtual ArraySegment<byte> SerializeByte(byte value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeByte(value));
        }

        protected virtual ArraySegment<byte> SerializeBoolean(bool value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeBoolean(value));
        }

        protected virtual ArraySegment<byte> SerializeInt32(Int32 value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeInt(value));
        }

        protected virtual ArraySegment<byte> SerializeInt64(Int64 value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeLong(value));
        }

        protected virtual ArraySegment<byte> SerializeDateTime(DateTime value)
        {
            var epochMilliseconds = (long)(value - EPOCH_START_DATETIME).TotalMilliseconds;
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeLong(epochMilliseconds));
        }

        protected virtual ArraySegment<byte> SerializeDouble(Double value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeLong(BitConverter.DoubleToInt64Bits(value)));
        }

        protected virtual ArraySegment<byte> SerializeSingle(Single value)
        {
            return new ArraySegment<byte>(_spyTranscoderUtil.EncodeLong(BitConverter.ToInt32(BitConverter.GetBytes(value), 0)));
        }

        protected virtual ArraySegment<byte> SerializeObject(object value)
        {
            using (var ms = new MemoryStream())
            {
                new BinaryFormatter().Serialize(ms, value);

                return new ArraySegment<byte>(ms.GetBuffer(), 0, (int)ms.Length);
            }
        }

        #endregion

        #region Typed deserialization

        protected virtual String DeserializeString(byte[] value)
        {
            //return Encoding.UTF8.GetString(value.Array, value.Offset, value.Count);
            return Encoding.UTF8.GetString(value);
        }

        protected virtual Boolean DeserializeBoolean(byte[] value)
        {
            return _spyTranscoderUtil.DecodeBoolean(value);
        }

        protected virtual Int32 DeserializeInt32(byte[] value)
        {
            return _spyTranscoderUtil.DecodeInt(value);
        }

        protected virtual Int64 DeserializeInt64(byte[] value)
        {
            return _spyTranscoderUtil.DecodeLong(value);
        }

        protected virtual DateTime DeserializeDateTime(byte[] value)
        {
            var epochMilliseconds = _spyTranscoderUtil.DecodeLong(value);
            return EPOCH_START_DATETIME.AddMilliseconds(epochMilliseconds);
        }

        protected virtual Double DeserializeDouble(byte[] value)
        {
            return BitConverter.Int64BitsToDouble(_spyTranscoderUtil.DecodeLong(value));
        }

        protected virtual Single DeserializeSingle(byte[] value)
        {
            byte[] bytes = BitConverter.GetBytes(_spyTranscoderUtil.DecodeInt(value));
            return BitConverter.ToSingle(bytes, 0);
        }

        protected virtual Byte DeserializeByte(byte[] data)
        {
            return _spyTranscoderUtil.DecodeByte(data);
        }

        protected virtual object DeserializeObject(byte[] value)
        {
            //using (var ms = new MemoryStream(value.Array, value.Offset, value.Count))
            using (var ms = new MemoryStream(value))
            {
                return new BinaryFormatter().Deserialize(ms);
            }
        }

        #endregion

        #region GZip
        private ArraySegment<byte> Compress(ArraySegment<byte> data)
        {
            using (var outStream = new MemoryStream())
            {
                using (var compressStream = new GZipStream(outStream, CompressionMode.Compress))
                {
                    using (var inStream = new MemoryStream(data.Array))
                    {
                        inStream.CopyTo(compressStream);
                        return new ArraySegment<byte>(outStream.ToArray());
                    }
                }
            }
        }
        private ArraySegment<byte> Decompress(ArraySegment<byte> data)
        {
            using (var inStream = new MemoryStream(data.Array))
            {
                using (var decompressStream = new GZipStream(inStream, CompressionMode.Decompress))
                {
                    using (var outStream = new MemoryStream())
                    {
                        decompressStream.CopyTo(outStream);
                        return new ArraySegment<byte>(outStream.ToArray());
                    }
                }
            }
        } 
        #endregion
    }

    internal class SpyMemcachedTranscoderUtils
    {

        private readonly bool _packZeros;

        public SpyMemcachedTranscoderUtils(bool pack = true)
        {
            _packZeros = pack;
        }

        public byte[] EncodeNum(long value, int maxBytes)
        {
            byte[] rv = new byte[maxBytes];
            for (int i = 0; i < rv.Length; i++)
            {
                int pos = rv.Length - i - 1;
                rv[pos] = (byte)((value >> (8 * i)) & 0xff);
            }
            if (_packZeros)
            {
                int firstNon0 = 0;
                // Just looking for what we can reduce
                while (firstNon0 < rv.Length && rv[firstNon0] == 0)
                {
                    firstNon0++;
                }
                if (firstNon0 > 0)
                {
                    byte[] tmp = new byte[rv.Length - firstNon0];
                    Array.Copy(rv, firstNon0, tmp, 0, rv.Length - firstNon0);
                    rv = tmp;
                }
            }
            return rv;
        }

        public byte[] EncodeLong(long value)
        {
            return EncodeNum(value, 8);
        }

        public long DecodeLong(byte[] value)
        {
            long rv = 0;
            foreach (byte i in value)
            {
                rv = (rv << 8) | (i < 0 ? 256 + i : i);
            }
            return rv;
        }

        public byte[] EncodeInt(int value)
        {
            return EncodeNum(value, 4);
        }

        public int DecodeInt(byte[] value)
        {
            if (value.Length > 4)
                throw new InvalidOperationException("Too long to be an int (" + value.Length + ") bytes");

            return (int)DecodeLong(value);
        }

        public byte[] EncodeByte(byte value)
        {
            return new byte[] { value };
        }

        public byte DecodeByte(byte[] value)
        {
            if (value.Length > 1)
                throw new InvalidOperationException("Too long for a byte");

            byte rv = 0;
            if (value.Length == 1)
            {
                rv = value[0];
            }
            return rv;
        }

        public byte[] EncodeBoolean(bool b)
        {
            byte[] rv = new byte[1];
            rv[0] = (byte)(b ? '1' : '0');
            return rv;
        }

        public bool DecodeBoolean(byte[] value)
        {
            if (value.Length != 1)
                throw new InvalidOperationException("Wrong length for a boolean");

            return value[0] == '1';
        }
    }

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM