HDInsight in Azure Data Factory Hive CREATE EXTERNAL TABLE…LOCATION: No FileSystem for scheme: adl

Question

I have a Data Factory Pipeline with an HD Insight Pipeline which is attempting to access an Azure Data Lake directory.

"type": "HDInsightHive",
            "typeProperties": {
                "scriptPath": "mpp-hive-scripts/parse_log.q",
                "scriptLinkedService": "AzureStorageLinkedService",
                "defines": {
                    "inputtable": "adl://mppmetering.azuredatalakestore.net/hour=23",
                    "outputtable": "wasb://mpp-metering-hive-output@mppmetering.blob.core.windows.net/output/"
                }
            },

I get the following error when I run this pipeline:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: ad

This works fine in a pipeline which accesses a Azure Blob store with the following spec:

"type": "HDInsightHive",
            "typeProperties": {
                "scriptPath": "mpp-hive-scripts/parse_log.q",
                "scriptLinkedService": "AzureStorageLinkedService",
                "defines": {
                    "inputtable": "wasb://mpp-metering-live@mppmetering.blob.core.windows.net/hour=23",
                    "outputtable": "wasb://mpp-metering-hive-output@mppmetering.blob.core.windows.net/output/"
                }
            },

Any help would be appreciated. js

Answer 1

Currently, Azure Data Factory doesn't support referring ADL Store in Hive scripts. The only supported store is blob storage (wasb). This is the reason why you are seeing the error. You need to use blob storage (wasb links) in your 'defines' section. In case the intent is to land the data to ADL store, you can use the 'Copy' activity in ADF to move the data from Blob store to ADL store.