json.net: how can I deserialize 'similar but different' external json structures to a single class

Question

I'm looking for a way, preferably using JSON.NET (using latest), to deserialize multiple external JSON formats/structures to a single class.

simplified example (the differences are greater than this but the different json's contain similar information):

external 1

{
    "id": 1234,
    "person": {
        "name": "john",
        "surname": "doe"           
    }
}

external 2

{
    "ref": "1234",
    "firstName": "JOHN",
    "lastName": "DOE"
}

internal (this is not real, it's just for show)

{
    "tag": "1234",
    "name1": "John",
    "name2": "Doe"
}

Is there some way/library which perhaps allows you to configure the mapping using a mapping.json file. Preferably one that also allows formatting of the values etc. These are only 2 examples, but we have many more.

Edit: We can tell/hint JSON.NET what source the given JSON is coming from. Therefor we don't have to have a single schema/contract/solution that can handle all different scenarios. We would actually prefer to have a .json mapping/transform config file for every different external json structure (to keep it future proof without the need of having to rebuild everything).

Edit 2: What I've now done is the following based on Pavel Baravik answer is to go through all properties of a 'schema/transformation' JSON. This JSON has the same structure as the final JSON of the object that we want to transform the original external JSON to. If a token is of type 'String' we'll parse that string (supports {{ }} and plain =) and use that as a path to pull values from the original external JSON. In the meantime the final JSON is being constructed, which afterwards will be deserialized to our internal object.
I think we could improve the performance of this code by 'sort-of' compiling it using an Expression tree.

static void Main(string[] args)
{
    var schema = @"
    {
        ""person"": {                    
            ""name"": ""=test.firstName"",
            ""fullName"": ""{{test.firstName}} {{lastName}}"",
            ""surName"":  ""=lastName""
        }
    }";

    var json1 = @"
    {
        ""test"": {
            ""firstName"": ""John""
        },                
        ""lastName"": ""Doe"",
    }";

    Console.WriteLine(Transform(json1, schema).ToString());
    Console.ReadLine();
}

public static JObject Transform(string json, string schema)
{
    var j = JObject.Parse(json);
    var s = JObject.Parse(schema);
    var t = Transform(s, j);

    return t;
}

public static JObject Transform(JObject schema, JObject source)
{
    var target = new JObject();

    foreach (var child in schema.Children())
    {
        var property = child as JProperty;
        if (property != null)
        {
            var schemaToken = property.Value;
            var allowClone = true;

            JToken t = null;
            if (schemaToken.Type == JTokenType.Object)
            {
                t = Transform((JObject) schemaToken, source);
            }
            else if (schemaToken.Type == JTokenType.String)
            {
                allowClone = false;
                t = TransformProperty(source, (JValue)schemaToken);
            }

            if (t != null || allowClone)
            {
                target.Add(property.Name, (t ?? property).DeepClone());
            }
        }
    }

    return target;
}

private static readonly Regex MoustacheRegex = new Regex(@"\{\{[^\}]+\}\}", RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.Singleline);

private static JToken TransformProperty(JObject source, JValue jstring)
{
    var str = (string)jstring.Value;
    JToken t = null;

    // if string starts with =
    if (str.StartsWith("="))
    {
        t = GetTokenByPath(source, str.Substring(1));
    }
    else
    {
        var allFound = true;

        str = MoustacheRegex.Replace(str, m =>
        {
            var mv = m.Value;
            var mt = GetTokenByPath(source, mv.Substring(2, mv.Length - 4));
            if (mt == null) allFound = false;

            return mt?.ToString() ?? string.Empty;
        });

        if (allFound)
            t = new JValue(str.Trim());
    }

    return t;
}

private static JToken GetTokenByPath(JObject source, string path)
{
    JToken t = null;
    var pathItems = path.Split('.');
    var s = source;

    for (var i = 0; i < pathItems.Length && s != null; ++i, s = t as JObject)
    {
        t = s[pathItems[i]];
    }

    return t;
}

EDIT: (nice JTransform class)

public class JTransform
{
    private InternalJTransform _internal;

    public void Load(string filePath)
    {
        using (var stream = File.OpenRead(filePath))
        using (var reader = new StreamReader(stream))
        {
            Load(new JsonTextReader(reader));
        }
    }

    public void Load(string filePath, Encoding encoding)
    {
        using (var stream = File.OpenRead(filePath))
        using (var reader = new StreamReader(stream, encoding))
        {
            Load(new JsonTextReader(reader));
        }
    }

    public void Load(JsonReader reader)
    {
        _internal = new InternalJTransform(reader);
    }

    public JObject Transform(JsonReader sourceReader)
    {
        return _internal.Transform(sourceReader);
    }

    public JObject Transform(JObject source)
    {
        return _internal.Transform(source);
    }

    public T TransformObject<T>(object obj)
    {
        return _internal.TransformObject<T>(obj);
    }

    public T TransformObject<T>(JObject source, JsonSerializer serializer = null)
    {
        return _internal.TransformObject<T>(source, serializer);
    }

    #region InternalJTransform

    private sealed class InternalJTransform
    {
        private static readonly Regex MoustacheRegex = new Regex(@"\{\{[^\}]+\}\}", RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.Singleline);

        private JsonSerializer _serializer;
        private JObject _template;
        private bool _ignoreUndefined;

        public InternalJTransform(JsonReader reader)
        {
            var json = JObject.Load(reader);

            _template = json["template"] as JObject;
            _serializer = new JsonSerializer();

            var settings = json["settings"];

            if (settings["camelCase"]?.Value<bool>() ?? false)
                _serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();

            if (settings["ignoreNull"]?.Value<bool>() ?? false)
                _serializer.NullValueHandling = NullValueHandling.Ignore;

            _ignoreUndefined = (settings["ignoreUndefined"]?.Value<bool>() ?? settings["ignoreNull"]?.Value<bool>() ?? false);
        }

        private void Load(JsonReader reader)
        {
            var json = JObject.Load(reader);

            var template = json["template"] as JObject;
            var serializer = new JsonSerializer();

            var settings = json["settings"];

            if (settings["camelCase"]?.Value<bool>() ?? false)
                serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();

            if (settings["ignoreNull"]?.Value<bool>() ?? false)
                serializer.NullValueHandling = NullValueHandling.Ignore;

            _ignoreUndefined = (settings["ignoreNull"]?.Value<bool>() ?? false);
            _serializer = serializer;
            _template = template;
        }

        public JObject Transform(JsonReader sourceReader)
        {
            var obj = JObject.Load(sourceReader);
            return TransformInternal(_template, obj, _serializer);
        }

        public JObject Transform(JObject source)
        {
            return TransformInternal(_template, source, _serializer);
        }

        public T TransformObject<T>(object obj)
        {
            var source = JObject.FromObject(obj);
            var im = TransformInternal(_template, source, _serializer);
            return im.ToObject<T>(_serializer);
        }

        public T TransformObject<T>(JObject source, JsonSerializer serializer = null)
        {
            var obj = TransformInternal(_template, source, _serializer);
            return obj.ToObject<T>(serializer ?? _serializer);
        }

        private JObject TransformInternal(JObject template, JObject source, JsonSerializer serializer)
        {
            var ignoreNull = serializer.NullValueHandling == NullValueHandling.Ignore;
            var target = new JObject();

            foreach (var property in template.Properties())
            {
                var token = property.Value;

                if (token.Type == JTokenType.Object)
                {
                    token = TransformInternal((JObject)token, source, serializer);
                }
                else if (token.Type == JTokenType.String)
                {
                    token = TransformStringToken(source, (JValue)token);

                    // handle undefined, not found, values
                    if (token == null && _ignoreUndefined) continue;
                }

                // handle real null values (this does not include null values set in the template)
                if (token != null && token.Type == JTokenType.Null && ignoreNull) continue;

                target.Add(property.Name, token?.DeepClone());
            }

            return target;
        }

        private JToken TransformStringToken(JObject source, JValue jstring)
        {
            var str = (string)jstring.Value;
            JToken t = null;

            // if string starts with =
            if (str.StartsWith("="))
            {
                t = GetTokenByPath(source, str.Substring(1));
            }
            else
            {
                var allFound = true;

                str = MoustacheRegex.Replace(str, m =>
                {
                    var mv = m.Value;
                    var mt = GetTokenByPath(source, mv.Substring(2, mv.Length - 4));
                    if (mt == null) allFound = false;

                    return mt?.ToString() ?? string.Empty;
                });

                if (allFound)
                    t = new JValue(str.Trim());
            }

            return t;
        }

        private static JToken GetTokenByPath(JObject source, string path)
        {
            JToken t = null;
            var pathItems = path.Split('.');
            var s = source;

            for (var i = 0; i < pathItems.Length && s != null; ++i, s = t as JObject)
            {
                t = s[pathItems[i]];
            }

            return t;
        }
    }

    #endregion
}

Answer 1

Take a look at the CustomCreationConverter in JSON.NET http://www.newtonsoft.com/json/help/html/CustomCreationConverter.htm you can make different converters and decide which one to use based on the JSON you have. they could all output the same class

Answer 2

You can firstly 'flatten' your input structures with use of JsonReader and then map to a single class (adopted from JSON.NET deserialize a specific property ).

void Main()
{
    var json0 = @"{
    ""id"": 1234,
    ""person"": {
        ""name"": ""john"",
        ""surname"": ""doe""           
    }";

    var json1 = @"  {
    ""ref"": ""1234"",
    ""firstName"": ""JOHN"",
    ""lastName"": ""DOE""
    }";

    foreach (var j in new []{json0, json1})
    {
        var name = GetFirstInstance<string>(new [] {"person.name", "firstName", "name1"}, j);
        var surname = GetFirstInstance<string> (new [] {"person.surname", "lastName", "name2"}, j);

        new {name, surname}.Dump();
    }
}

public T GetFirstInstance<T>(string[] path, string json)
{
    using (var stringReader = new StringReader(json))
    using (var jsonReader = new JsonTextReader(stringReader))
    {
        while (jsonReader.Read())
        {
            if (jsonReader.TokenType == JsonToken.PropertyName  && path.Contains((string)jsonReader.Path))
            {
                jsonReader.Read();

                var serializer = new JsonSerializer();
                return serializer.Deserialize<T>(jsonReader);
            }
        }
        return default(T);
    }
}