使用SSIS脚本组件读取Azure Data Lake Store文件

Question

Appreciate your suggestions.
My Requirement is, Read json file from ADLS using SSIS and load into SQL table

Implementation: I have implemented the code to read json file content in .Net Console app. This is working fine in Console app. I copied the same code in SSIS Script component, but it throws "The type initializer for 'Microsoft.Azure.DataLake.Store.AdlsClient' threw an exception" exception in AdlsClient.CreateClient.

using Microsoft.Rest;
using Microsoft.Rest.Azure.Authentication;
using Microsoft.Azure.Management.DataLake.Store;
using Microsoft.Azure.DataLake.Store;
using Microsoft.Azure.DataLake.Store.AclTools;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

JObject results = new JObject();
        string applicationId = "<appid>;
        string secretKey = <secretekey>;
        string tenantId = <tenantid>;
        string adlsAccountName = "<ADLSNAME>.azuredatalakestore.net";
        ServiceClientCredentials creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;

AdlsClient adlsClient = AdlsClient.CreateClient(adlsAccountName, creds);
string srcPath = @"/InputFiles/1636274001230002_20180621_104427.json";
using (StreamReader readStream = new 
StreamReader(adlsClient.GetReadStream(srcPath)))
        {
            var p2Object = JsonConvert.DeserializeObject(readStream.ReadToEnd());
            results = JObject.Parse(p2Object.ToString());
        }

        date = ((string)results["eeData"][0]["startDate"]);
        machine = ((string)results["eeData"][0]["machineName"]);
        ppl = ((string)results["eeData"][0]["ppl"]);

Answer 1

The issue is with the reference path missing in SSIS Script component for the 3rd party DLLs. In Console App I am able to install NuGet package manager. But in SSIS Script component, the NuGet package installation is failed and SSIS component is missing the reference. The below code will force the script component compiler to refer the DLLs from the given path.

Add this code above PreExecute() / Main() method.

static ScriptMain()
    {
        AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(CurrentDomain_AssemblyResolve);
    }

    static System.Reflection.Assembly CurrentDomain_AssemblyResolve(object sender, ResolveEventArgs args)
    {
        if (args.Name.Contains("Newtonsoft.Json"))
        {
            return System.Reflection.Assembly.LoadFile(@"C:\Program Files\Microsoft SDKs\Azure\.NET SDK\v2.9\ToolsRef\Newtonsoft.Json.dll");
        }

        if (args.Name.Contains("Microsoft.Azure.DataLake.Store"))
        {
            return System.Reflection.Assembly.LoadFile(@"C:\Program Files\WindowsPowerShell\Modules\AzureRM.DataLakeStore\5.2.0\Microsoft.Azure.DataLake.Store.dll");
        }

        if (args.Name.Contains("Microsoft.Rest.ClientRuntime.Azure.Authentication"))
        {
            return System.Reflection.Assembly.LoadFile(@"C:\Program Files\WindowsPowerShell\Modules\Azure\5.1.2\StorSimple\Microsoft.Rest.ClientRuntime.Azure.Authentication.dll");
        }

        if (args.Name.Contains("Microsoft.Rest.ClientRuntime"))
        {
            return System.Reflection.Assembly.LoadFile(@"C:\Program Files\WindowsPowerShell\Modules\Azure\5.1.2\Services\Microsoft.Rest.ClientRuntime.dll");
        }
       if (args.Name.Contains("NLog"))
        {
        return System.Reflection.Assembly.LoadFile(@"C:\Users\<user>\source\repos\Integration Services Project2\NLog.dll");
        }

        return null;

       }