简体繁体 English

在reduce阶段工作时如何开始map阶段

[英]How to start the map phase while the reduce phase is working

原文 2012-08-17 16:03:11 2 1 java/ hadoop/ reduce/ mapper

I have this scenario. 我有这种情况。 JobA and JobB. JobA和JobB。 Is there a chance to start the JobB mapping phase using the data provided by the JobA reduce phase while this is still working? 在仍然有效的情况下，是否有机会使用JobA reduce阶段提供的数据来启动JobB映射阶段？

Thanks! 谢谢！

1 个解决方案

The only thing that comes in my mind is to have a thread(started in your driver class) which permanently checks the output directory of JobA. 我唯一想到的就是拥有一个线程（从驱动程序类开始），该线程可以永久检查JobA的输出目录。 When a particular(set of) part-r-xxxx file(s) were created and completely written you can start JobB having as input that particular(set of) part-r-xxxx file(s). 创建并完全编写了一部分特定的r-xxxx文件后，您可以启动JobB，并输入该特定部分r-xxxx的文件。

The only problem I can identify now is the one related to checking whether a part-r-xxxx file was completely written. 我现在可以确定的唯一问题是与检查part-r-xxxx文件是否已完全写入有关的问题。

在hadoop的地图阶段之后，reduce阶段如何工作 - How reduce phase works after map phase in hadoop

Hadoop Map Reduce CustomRecordReader缩减阶段 - Hadoop Map Reduce CustomRecordReader Reduce Phase

MapReduce 职位：我怎么收<text, intwritable>在 Map 阶段和 output<text, text> 在减少阶段？</text,></text,> - MapReduce Job: How do I take in <Text, IntWritable> during Map phase and output <Text, Text> in Reduce phase?

MongoDB的还原阶段未按预期工作 - MongoDB's reduce-phase is not working as expected

如何衡量Hadoop中每个阶段（映射，随机播放/排序，缩小）的持续时间？ - How can I measure the duration of each phase(map, shuffle/sort, reduce) in Hadoop?

Hadoop：测量每个阶段的持续时间（映射，随机/排序，减少） - Hadoop: Measuring the duration of each phase(map, shuffle/sort, reduce)

Hadoop完全跳过了减少阶段 - Hadoop is skipping reduce phase entirely

在还原阶段更改密钥 - Changing key during reduce phase

如何在Xvfb上启动mvn测试阶段？ - How do I start mvn test phase on Xvfb?

Hadoop：Reduce 阶段开始：FileNotFoundException 输出/file.out.index 不存在 - Hadoop: Start of Reduce phase: FileNotFoundException output/file.out.index does not exist

暂无

暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 在hadoop的地图阶段之后，reduce阶段如何工作 - How reduce phase works after map phase in hadoop Hadoop Map Reduce CustomRecordReader缩减阶段 - Hadoop Map Reduce CustomRecordReader Reduce Phase MapReduce 职位：我怎么收<text, intwritable>在 Map 阶段和 output<text, text> 在减少阶段？</text,></text,> - MapReduce Job: How do I take in <Text, IntWritable> during Map phase and output <Text, Text> in Reduce phase? MongoDB的还原阶段未按预期工作 - MongoDB's reduce-phase is not working as expected 如何衡量Hadoop中每个阶段（映射，随机播放/排序，缩小）的持续时间？ - How can I measure the duration of each phase(map, shuffle/sort, reduce) in Hadoop? Hadoop：测量每个阶段的持续时间（映射，随机/排序，减少） - Hadoop: Measuring the duration of each phase(map, shuffle/sort, reduce) Hadoop完全跳过了减少阶段 - Hadoop is skipping reduce phase entirely 在还原阶段更改密钥 - Changing key during reduce phase 如何在Xvfb上启动mvn测试阶段？ - How do I start mvn test phase on Xvfb? Hadoop：Reduce 阶段开始：FileNotFoundException 输出/file.out.index 不存在 - Hadoop: Start of Reduce phase: FileNotFoundException output/file.out.index does not exist

相关标签

粤ICP备18138465号 © 2020-2024 STACKOOM.COM