I'm writing a program for a daily upload to s3 of all our hive tables from a particular db. This database contains records from many years ago, howeve ...

I am trying to move data in s3 which is partitioned on a date string at rest(source) to another location where it is partitioned at rest (destination) ...

I am using S3DistCp on an EMR cluster in order to aggregate around 200K small files (for a total of 3.4GB) from a S3 bucket to another path in the sam ...

I have csv files in lzo format in HDFS I would like to load these files in to s3 and then to snowflake, as snowflake does not provides lzo compression ...

I'd like to copy some files from emr-hdfs to s3 bucket using s3-dist-cp, I've tried this cmd from "EMR Master Node": this command executes fine but ...

My spark application running on AWS EMR loads data from JSON array stored in S3. The Dataframe created from it is then processed via Spark engine. My ...

When I running command below, I got a error about auxService. In many QnA, I found a solution like this link. But there is no process for node ...

I was looking into the documentation of s3distcp (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html) but I was not able to fi ...

I'm working on an AWS-EMR cluster and added a step to run S3DISTCP (https://docs.aws.amazon.com/es_es/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html), ...

I'm trying to copy data from an EMR cluster to S3 using s3-distcp. Can I specify the number of reducers to a greater value than the default so as to f ...

On an EMR, I am using s3-dist-cp --groupBy in order to name the file with random fileName in a folder to a name that i wish to rename it to in S3: s3 ...

i am trying to copy data from one hdfs cluster to another using distcp command.following is the command which i submitted hadoop distcp hdfs://sourc ...

I have a jar file that is being provided to spark-submit.With in the method in a jar. I’m trying to do a I also installed s3-dist-cp on all salves ...

I want to upload few files into AWS bucket from hadoop. I have AWS ACCESS KEY, SECRET KEY and S3 IMPORT PATH. I am not able to access though AWS CLI ...

Can someone please help me with authentication while moving the data from hdfs to S3. To connect to S3, I am generating session based credentials usin ...

I have requirement where I need to copy files from one S3 bucket to other S3 bucket. These buckets are present in different AWS account. I tried using ...

I am copying 800 avro files, size around 136 MB, from HDFS to S3 on EMR cluster, but Im getting this exception: The configuration for the EMR clust ...

I am using s3distcp to copy a 500GB dataset into my EMR cluster. It's a 12 node r4.4xlarge cluster each with 750GB disk. It's using the EMR release la ...

Trying to figure out how to export data from HDFS which is outputted by Apache Spark Streaming job. Following diagram defines solution architecture: ...

I have a spark job where I have huge file as output 300 gb to S3 . My requirement is to rename all part files and then we have to move to final folder ...