“分区数据”是什么意思 - S3

Question

I want to use Netflix's outputCommitter (Using Spark with Amazon EMR). In the README there are 2 options:

1. S3DirectoryOutputCommitter - for writing unpartitioned data to S3 with conflict resolution.
2. S3PartitionedOutputCommitter - for writing partitioned data to S3 with conflict resolution.

I tried to understand the differences but unsuccessfully. Can someone explain what is "partitioned data" in s3?

Answer 1

according to the hadoop docs, "This committer an extension of the “Directory” committer which has a special conflict resolution policy designed to support operations which insert new data into a directory tree structured using Hive's partitioning strategy: different levels of the tree represent different columns."

search in the hadoop docs for the full details.

be aware that the EMR committers are not the ASF S3A ones, so take different config options and have their own docs. but since their work is a reimplementation of the.netflix work, they should do the same thing here

Answer 2

I'm not familiar with outputCommitter , by partitioned data in Amazon S3 normally refers to splitting files amongst directories to reduce the amount of data that needs to be read from disk.

For example:

/data/month=1/
/data/month=2/
/data/month=3/
...

If a Hive-type query is run against the data with a clause like WHERE month=1 , then it would only need to look in the month=1/ subdirectory, thereby saving 2/3rds of disk access.