JSON Serialization in C# SSIS

Question

RESOLVED: I stopped treating the string like an array and instead added a line-feed delimiter to the connection manager, so that each row was treated like a new line. This has resolved the issue and eliminated the need to handle this via a C# script.

I've built an SSIS package that uses a C# script task to deserialize JSON strings and insert them into a table.

I've got this working with a prototype JSON string:

{"name":"Test 1","code":"398057008","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"410605003","prefix":"[C] "}],"prefix":"[C] "}

But when I try to add a second item to the JSON string I receive an error:

[{"name":"Test 2","code":"398057008","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"410605003","prefix":"[C] "}],"prefix":"[C] "},{"name":"Test 2","code":"44255352","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"53252355","prefix":"[C] "}],"prefix":"[C] "}]

Error:

Type 'SC_8aae662509ae4bab8491323924579173.Diagnosis' is not supported for deserialization of an array.

From what I can understand, this is simply saying that my 'Parser' essentially cannot support an array of JSON fields in a string.

Here is my main code block:

#region Namespaces
using System;
using System.Data;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Web.Script.Serialization;
using Microsoft.SqlServer.Dts.Pipeline;

namespace SC_8aae662509ae4bab8491323924579173
#endregion
{

    [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
    public class ScriptMain : UserComponent
    {

        ///<param name="Row">The row that is currently passing through the component</param>;
        public override void Input0_ProcessInputRow(Input0Buffer Row)
        {
            JavaScriptSerializer js = new JavaScriptSerializer();

            // Give the input column a variable to make it easier to reference.
            BlobColumn combinedColumn = Row.Column0;

            // Convert from blob to string
            string reviewConverted = System.Text.Encoding.ASCII.GetString(combinedColumn.GetBlobData(0, Convert.ToInt32(combinedColumn.Length)));

            Diagnosis diagnosis = new Diagnosis();

            // Deserialize the string
            diagnosis = js.Deserialize<Diagnosis>(reviewConverted);

            // Assign values to output columns
            Row.name = diagnosis.name;
            Row.code = diagnosis.code;
            Row.table = diagnosis.table;
            Row.addedby = diagnosis.addedby;
            Row.dateadded = diagnosis.dateadded;

            Row.qualifierName = diagnosis.Qualifier[0].name;
            Row.qualifierValue = diagnosis.Qualifier[0].value;
            Row.qualifierCode = diagnosis.Qualifier[0].code;
            Row.qualifierPrefix = diagnosis.Qualifier[0].prefix;           

            if (diagnosis.Qualifier.Length == 2)
            {
                Row.lrName = diagnosis.Qualifier[1].name;
                Row.lrValue = diagnosis.Qualifier[1].value;
                Row.lrCode = diagnosis.Qualifier[1].code;
                Row.lrSuffix = diagnosis.Qualifier[1].prefix;
            }

            Row.jsonString = reviewConverted;

            Row.prefix = diagnosis.prefix;
            Row.suffix = diagnosis.suffix;

        }
    }
}

Qualifier class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SC_8aae662509ae4bab8491323924579173
{
    class qualifier
    {
        public string name { get; set; }
        public string value { get; set; }
        public string code { get; set; }
        public string prefix { get; set; }
    }
}

Diagnosis class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace SC_8aae662509ae4bab8491323924579173
{
    class Diagnosis
    {
        public string name { get; set; }
        public string code { get; set; }
        public string table { get; set; }
        public string addedby { get; set; }
        public string dateadded { get; set; }
        public qualifier[] Qualifier { get; set; }
        public string prefix { get; set; }
        public string suffix { get; set; }
        public string jsonString { get; set; }
    }
}

I've tried to cater for more than one diagnosis with an array and my if function, but to no avail. Any ideas? Many thanks.

Answer 1

You say, that you pass array instead of an object in json. So why do you keep trying to deserialize into Diagnosis object? You should really use Collection with your type Diagnosis here:

// Deserialize the string
var diagnosisCollection = js.Deserialize<ICollection<Diagnosis>>(reviewConverted);

instead of old

// Deserialize the string
diagnosis = js.Deserialize<Diagnosis>(reviewConverted);

And then assign values for you output columns, using foreach

foreach (var diagnosis in diagnosisCollection ) {
    // do stuff with your buffer here

}

Answer 2

SSIS script can be a wierd environment. And you're right to avoid using a 3rd party JSON parser here. But somewhere you've got the type definitions messed up.

I often find it useful to write, test and troubleshoot code in a console app before integrating it into an SSIS script component/task. EG

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp14
{
    class Program
    {
        class qualifier
        {
            public string name { get; set; }
            public string value { get; set; }
            public string code { get; set; }
            public string prefix { get; set; }
        }

        class Diagnosis
        {
            public string name { get; set; }
            public string code { get; set; }
            public string table { get; set; }
            public string addedby { get; set; }
            public string dateadded { get; set; }
            public qualifier[] Qualifier { get; set; }
            public string prefix { get; set; }
            public string suffix { get; set; }
            public string jsonString { get; set; }
        }
        static void Main(string[] args)
        {
            var json = @"
[
    {
        ""name"": ""Test 2"",
        ""code"": ""398057008"",
        ""table"": ""SNOMEDCT"",
        ""addedby"": ""morgan.baxter"",
        ""dateadded"": 1544523489235,
        ""qualifier"": [
            {
                ""name"": ""Qualifier"",
                ""value"": ""Confirmed Diagnosis"",
                ""code"": ""410605003"",
                ""prefix"": ""[C] ""
            }
        ],
        ""prefix"": ""[C] ""
    },
    {
        ""name"": ""Test 2"",
        ""code"": ""44255352"",
        ""table"": ""SNOMEDCT"",
        ""addedby"": ""morgan.baxter"",
        ""dateadded"": 1544523489235,
        ""qualifier"": [
            {
                ""name"": ""Qualifier"",
                ""value"": ""Confirmed Diagnosis"",
                ""code"": ""53252355"",
                ""prefix"": ""[C] ""
            }
        ],
        ""prefix"": ""[C] ""
    }
]
";
            System.Web.Script.Serialization.JavaScriptSerializer js = new System.Web.Script.Serialization.JavaScriptSerializer();

            // Deserialize the string
            var diagnoses = js.Deserialize<Diagnosis[]>(json);

            Console.WriteLine("Complete");
            Console.ReadKey();


        }
    }
}