如何使用 Sqoop 通过 Docker 容器将导入数据从 SQL 服务器连接到 HDFS？

Question

I have configured two docker containers on the google compute engine.

First container: I have loaded a database on it and it runs well when I connect it via a SQL client on the localhost/mylaptop. Note I am also using the --network so I can talk between two containers.

# Run the docker container 
sudo docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=mypassword" \
   -p 1433:1433 --name data-engr-sql-svr -h data-engr-sql-svr \
   --network nc-de-network \
   -d mcr.microsoft.com/mssql/server:2019-latest

Second container: I am running a custom container that has Cloudera express installed on it.

It works wells the configuration is as follows. I have configured to run this on system startup, and I have to ssh into port 122 to use the container.

docker run -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-h cnt7-xxx-cdh63 \
--name cnt7-xxx-cdh63 \
--network nc-de-network \
-p 122:22 \
-p 7180:7180 \
-p 8889:8889 \
-p 3306:3306 \
-p 8890:8890 \
-p 4040:4040 \
-p 18088:18088 \
-p 10000:10000 \
-p 21050:21050 \
-p 9870:9870 \
-p 9092:9092 \
-p 2181:2181 \
-p 11000:11000 \
-p 41414:41414 \
-p 8088:8088 \
--privileged=true \
-it cnt7-xxx-cdh63 /usr/sbin/init

Import from SQL Server database to HDFS. I tried to use the following.

sudo -u hdfs sqoop list-databases \
--connect "jdbc:sqlserver://localhost:1433;instanceName=data-engr-sql-svr;databaseName=AdventureWorks2019" \
--username sa \
--password pwd

But I get the following error

com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host localhost, port 1433 has failed. Error: "Connection refused. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".

How do I make this work? Note: Both the containers individually work fine

Answer 1

You need to use data-engr-sql-svr:1433 , not localhost as the connection string.

Note: you don't need a whole Cloudera environment to run sqoop, and you should use docker exec rather than ssh