How can I serialize a Newtonsoft JToken to JSON using System.Text.Json?

Question

In the process of upgrading to ASP.NET Core 5, we have encountered a situation where we need to serialize and return a Json.NET JObject (returned by some legacy code we can't yet change) using System.Text.Json . How can this be done in a reasonably efficient manner, without re-serializing and re-parsing the JSON to a JsonDocument or reverting back to Json.NET completely via AddNewtonsoftJson() ?

Specifically, say we have the following legacy data model:

public class Model
{
    public JObject Data { get; set; }
}

When we return this from ASP.NET Core 5.0, the contents of the "value" property get mangled into a series of empty arrays. Eg:

var inputJson = @"{""value"":[[null,true,false,1010101,1010101.10101,""hello"",""𩸽"",""\uD867\uDE3D"",""2009-02-15T00:00:00Z"",""\uD867\uDE3D\u0022\\/\b\f\n\r\t\u0121""]]}";
var model = new Model { Data = JObject.Parse(inputJson) };
var outputJson = JsonSerializer.Serialize(model);

Console.WriteLine(outputJson);

Assert.IsTrue(JToken.DeepEquals(JToken.Parse(inputJson), JToken.Parse(outputJson)[nameof(Model.Data)]));

Fails, and generates the following incorrect JSON:

{"Data":{"value":[[[],[],[],[],[],[],[],[],[],[]]]}}

How can I correctly serialize the JObject property with System.Text.Json ? Note that the JObject can be fairly large so we would prefer to stream it out rather than format it to a string and parse it again from scratch into a JsonDocument simply to return it.

Answer 1

It is necessary to create a custom JsonConverterFactory to serialize a Json.NET JToken hierarchy to JSON using System.Text.Json .

Since the question seeks to avoid re-serializing the entire JObject to JSON just to parse it again using System.Text.Json , the following converter descends the token hierarchy recursively writing each individual value out to the Utf8JsonWriter :

using System.Text.Json;
using System.Text.Json.Serialization;
using Newtonsoft.Json.Linq;

public class JTokenConverterFactory : JsonConverterFactory
{
    // In case you need to set FloatParseHandling or DateFormatHandling
    readonly Newtonsoft.Json.JsonSerializerSettings settings;
    
    public JTokenConverterFactory() { }

    public JTokenConverterFactory(Newtonsoft.Json.JsonSerializerSettings settings) => this.settings = settings;

    public override bool CanConvert(Type typeToConvert) => typeof(JToken).IsAssignableFrom(typeToConvert);

    public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        var converterType = typeof(JTokenConverter<>).MakeGenericType(new [] { typeToConvert} );
        return (JsonConverter)Activator.CreateInstance(converterType, new object [] { options, settings } );
    }

    class JTokenConverter<TJToken> : JsonConverter<TJToken> where TJToken : JToken
    {
        readonly JsonConverter<bool> boolConverter;
        readonly JsonConverter<long> longConverter;
        readonly JsonConverter<double> doubleConverter;
        readonly JsonConverter<decimal> decimalConverter;
        readonly JsonConverter<string> stringConverter;
        readonly JsonConverter<DateTime> dateTimeConverter;
        readonly Newtonsoft.Json.JsonSerializerSettings settings;

        public override bool CanConvert(Type typeToConvert) => typeof(TJToken).IsAssignableFrom(typeToConvert);

        public JTokenConverter(JsonSerializerOptions options, Newtonsoft.Json.JsonSerializerSettings settings)
        {
            // Cache some converters for efficiency
            boolConverter = (JsonConverter<bool>)options.GetConverter(typeof(bool));
            stringConverter = (JsonConverter<string>)options.GetConverter(typeof(string));
            longConverter = (JsonConverter<long>)options.GetConverter(typeof(long));
            decimalConverter = (JsonConverter<decimal>)options.GetConverter(typeof(decimal));
            doubleConverter = (JsonConverter<double>)options.GetConverter(typeof(double));
            dateTimeConverter = (JsonConverter<DateTime>)options.GetConverter(typeof(DateTime));
            this.settings = settings;
        }

        public override TJToken Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            // This could be substantially optimized for memory use by creating code to read from a Utf8JsonReader and write to a JsonWriter (specifically a JTokenWriter).
            // We could just write the JsonDocument to a string, but System.Text.Json works more efficiently with UTF8 byte streams so write to one of those instead.
            using var doc = JsonDocument.ParseValue(ref reader);
            using var ms = new MemoryStream();
            using (var writer = new Utf8JsonWriter(ms))
                doc.WriteTo(writer);
            ms.Position = 0;
            using (var sw = new StreamReader(ms))
            using (var jw = new Newtonsoft.Json.JsonTextReader(sw))
            {
                return Newtonsoft.Json.JsonSerializer.CreateDefault(settings).Deserialize<TJToken>(jw);
            }
        }

        public override void Write(Utf8JsonWriter writer, TJToken value, JsonSerializerOptions options) =>
            // Optimize for memory use by descending the JToken hierarchy and writing each one out, rather than formatting to a string, parsing to a `JsonDocument`, then writing that.
            WriteCore(writer, value, options);

        void WriteCore(Utf8JsonWriter writer, JToken value, JsonSerializerOptions options)
        {
            if (value == null || value.Type == JTokenType.Null)
            {
                writer.WriteNullValue();
                return;
            }

            switch (value)
            {
                case JValue jvalue when jvalue.GetType() != typeof(JValue): // JRaw, maybe others
                default: // etc
                    {
                        // We could just format the JToken to a string, but System.Text.Json works more efficiently with UTF8 byte streams so write to one of those instead.
                        using var ms = new MemoryStream();
                        using (var tw = new StreamWriter(ms, leaveOpen : true))
                        using (var jw = new Newtonsoft.Json.JsonTextWriter(tw))
                        {
                            value.WriteTo(jw);
                        }
                        ms.Position = 0;
                        using var doc = JsonDocument.Parse(ms);
                        doc.WriteTo(writer);
                    }
                    break;
                // Hardcode some standard cases for efficiency
                case JValue jvalue when jvalue.Value is bool v:
                    boolConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue when jvalue.Value is string v:
                    stringConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue when jvalue.Value is long v:
                    longConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue when jvalue.Value is decimal v:
                    decimalConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue when jvalue.Value is double v:
                    doubleConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue when jvalue.Value is DateTime v:
                    dateTimeConverter.WriteOrSerialize(writer, v, options);
                    break;
                case JValue jvalue:
                    JsonSerializer.Serialize(writer, jvalue.Value, options);
                    break;
                case JArray array:
                    {
                        writer.WriteStartArray();
                        foreach (var item in array)
                            WriteCore(writer, item, options);
                        writer.WriteEndArray();
                    }
                    break;
                case JObject obj:
                    {
                        writer.WriteStartObject();
                        foreach (var p in obj.Properties())
                        {
                            writer.WritePropertyName(p.Name);
                            WriteCore(writer, p.Value, options);
                        }
                        writer.WriteEndObject();
                    }
                    break;
            }
        }
    }
}

public static class JsonExtensions
{
    public static void WriteOrSerialize<T>(this JsonConverter<T> converter, Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        if (converter != null)
            converter.Write(writer, value, options);
        else
            JsonSerializer.Serialize(writer, value, options);
    }
}

Then the unit test in the question should be modified to use the following JsonSerializerOptions :

var options = new JsonSerializerOptions
{
    Converters = { new JTokenConverterFactory() },
};
var outputJson = JsonSerializer.Serialize(model, options);

Notes:

• The converter implements deserialization of JToken types as well as serialization, however since that wasn't a strict requirement of the question, it simply reads the entire JSON hierarchy into a JsonDocument , outputs it to a MemoryStream and re-parses it using Json.NET.

• Newtonsoft's JsonSerializerSettings may be passed to customize settings such as FloatParseHandling or DateFormatHandling during deserialization.

• To add JTokenConverterFactory to the ASP.NET Core serialization options, see Configure System.Text.Json-based formatters .

Demo fiddle with some basic tests here: fiddle #1 .

A prototype version that implements deserialization by streaming from a Utf8JsonReader to a JsonWriter without loading the entire JSON value into a JsonDocument can be found here: fiddle #2 .