[英]Is Hadoop's TooRunner thread-safe?
I would like to trigger a few Hadoop jobs simultaneously. 我想同时触发一些Hadoop作业。 I've created a pool of threads using Executors.newFixedThreadPool. 我使用Executors.newFixedThreadPool创建了一个线程池。 Idea is that if the pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time using 'ToolRunner.run'. 想法是,如果池大小为2,我的代码将使用'ToolRunner.run'在相同的时间触发2个Hadoop作业。 In my testing, I noticed that these 2 threads keep stepping on each other. 在我的测试中,我注意到这两个线程互相踩踏。
When I looked under the hood, I noticed that ToolRunner creates GenericOptionsParser which in turn calls a static method 'buildGeneralOptions'. 当我看到底层时,我注意到ToolRunner创建了GenericOptionsParser,后者又调用静态方法'buildGeneralOptions'。 This method uses 'OptionBuilder.withArgName' which uses an instance variable called, 'argName'. 此方法使用'OptionBuilder.withArgName',它使用名为'argName'的实例变量。 This doesn't look thread safe to me and I believe is the root cause of issues I am running into. 这对我来说看起来并不安全,我相信这是我遇到问题的根本原因。
Any thoughts? 有什么想法吗?
Confirmed that ToolRunner is NOT thread-safe: 确认ToolRunner不是线程安全的:
Original code (which runs into problems): 原始代码(遇到问题):
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
//set the configuration back, so that Tool can configure itself
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
} }
New Code(which works): 新规范(有效):
public static int run(Configuration conf, Tool tool, String[] args)
throws Exception{
if(conf == null) {
conf = new Configuration();
}
GenericOptionsParser parser = getParser(conf, args);
tool.setConf(conf);
//get the args w/o generic hadoop args
String[] toolArgs = parser.getRemainingArgs();
return tool.run(toolArgs);
}
private static synchronized GenericOptionsParser getParser(Configuration conf, String[] args) throws Exception {
return new GenericOptionsParser(conf, args);
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.