Sqoop export from hive to sql is stuck

Question

I'm using Windows Azure HDInsights Hadoop cluster and I am trying to export hive table from there to SQL Azure server.

The hive table is really simple (actually, for now it consists from one column data without any space characters and other special symbols). On the server side, it's also a table with the following schema:

CREATE TABLE exp (line [nvarchar] (100))

For the exporting, I am using the following PS script:

$tableName = 'exp'

$connectionString = "jdbc:sqlserver://$sqlDatabaseServerName.database.windows.net;user=$sqlDatabaseLogin@$sqlDatabaseServerName;password=$sqlDatabasePassword;database=$databaseName"

$exportDir = "wasb://test1@mystorage.blob.core.windows.net/"

$sqoopDef = New-AzureHDInsightSqoopJobDefinition -Command "export --connect $connectionString --table $tableName --export-dir $exportDir --fields-terminated-by '\t' --lines-terminated-by '\n'"

$sqoopJob = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $sqoopDef -Debug -Verbose
Wait-AzureHDInsightJob -WaitTimeoutInSeconds 3600 -Job $sqoopJob

Write-Host "Standard Error" -BackgroundColor Green
Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $sqoopJob.JobId -StandardError
Write-Host "Standard Output" -BackgroundColor Green
Get-AzureHDInsightJobOutput -Cluster $clusterName ;-JobId $sqoopJob.JobId -StandardOutput

For some reasons, exporting is stuck after 100% map and finishing after timeout without any errors and exceptions.

For now I know that job sees the data in Hadoop (if I specify other path, it throws exception). If hive data schema and sql table schema are mismatching, I also have exception. From SQL side I see the evidence of successful connections.

And yes, I have configured SQL server firewall to allow connection and Windows Azure services are allowed as well.

Does anybody have suggestion about what to do next? Would appreciate any help.

Answer 1

So far, an issue was solved by creating of clustered index for SQL table:

create clustered index exp_clustered_index on exp(line);

Seems like Azure SQL prohibits data upload without it.