Sqoop + S3 + Parquet导致错误的FS错误

Question

When trying to import data to S3 in Parquet format using Sqoop, as follows:

bin/sqoop import --connect 'jdbc:[conn_string]' --table [table] --target-dir s3a://bucket-name/ --hive-drop-import-delims --as-parquetfile

... I get the following error:

ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://bucket-name/, expected: hdfs://localhost:9000

I have no problem importing non-parquet data or working with s3a directly through HDFS. Seems like this issue , but it was supposedly fixed many versions ago.

Answer 1

Sqoop以木地板格式导入当前不支持将s3作为目标目录，可以使用的解决方法是将数据导入hdfs，然后使用distcp将hdfs复制到s3

Answer 2

如果将S3用作目标目录，则Sqoop不支持导入到镶木地板文件。