简体繁体 English

使用开源/免费软件批处理作业依赖性

[英]Batch Job Dependencies Using Open Source/Free Software

原文 2009-11-10 13:38:14 4 4 linux/ unix/ dependencies/ cron

I run a large data warehouse plant where we have a lot of nightly jobs running concerruently however many have dependencies on a extract or data load process before they start. 我在一个大型数据仓库工厂中工作，我们每天晚上都有很多工作同步进行，但是许多工作在开始之前就依赖于提取或数据加载过程。 Currently we use an 'expensive scheduling system' to scehdule these at the moment. 目前，我们目前使用“昂贵的调度系统”来计划这些。

Is there any way you can setup job dependencies using an open source or free unix/linux tool such as cron? 有什么方法可以使用开源或免费的unix / linux工具（例如cron）来设置作业依赖关系？

Moving to an open soruce solution would be great and save us lots! 改用开放式的解决方案将是一件好事，可以为我们节省很多！

Regards Matt 关于马特

4 个解决方案

Cfengine can be made to do something like this. 可以使Cfengine做这样的事情。 You can set it up as a cron replacement, running arbitrary commands at scheduled times, and you can set up "classes" so that certain actions are performed only if certain classes are enabled. 您可以将其设置为cron替代品，在计划的时间运行任意命令，还可以设置“类”，以便仅在启用某些类后才执行某些操作。 Classes can be anything from "this is a Linux system" to "it's currently between 5 and 10 minutes after the hour" to "system load is above value x" to "this arbitrary shell command that I just specified returned true", so you could set up your classes to indicate your job dependencies. 从“这是一个Linux系统”到“当前在5小时到10分钟之间”到“系统负载高于值x”再到“我刚刚指定的此任意shell命令返回true”之类的类都可以，所以您可以设置您的类以指示您的工作依赖性。

I doubt that this would be as powerful as a scheduling system (dependencies would have to be set up manually by configuring classes, scheduling concurrently would requires extra scripting or configuration work), but it is free and open source. 我怀疑它是否像调度系统一样强大（必须通过配置类来手动设置依赖关系，同时调度将需要额外的脚本或配置工作），但是它是免费的且开源的。

Version 2 of Cfengine was not particularly pleasant to work with (in the words of Seth Vidal , "it's [sic] syntax kills kittens"). Cfengine的版本2使用起来并不是特别令人愉快（用Seth Vidal的话说，“它的[原文]语法杀死了小猫”）。 I haven't used Cfengine 3. Puppet has similar design goals as Cfengine and may be easier to work with. 我没有使用Cfengine的3 木偶也有类似的设计目标为Cfengine的，并且可以更容易使用。

I asked a similar question last year (maybe Serverfault would be a better place these days?). 去年我问了一个类似的问题（也许Serverfault现在会是一个更好的地方？）。 There doesn't seem to be a simple, install-and-go solution unfortunately. 不幸的是，似乎没有一个简单的即装即用的解决方案。

Cron doesn't handle this natively. Cron本身无法处理此问题。 Can the process that loads the data write out a status file upon completion? 加载数据的过程是否可以在完成后写出状态文件？ This would allow subsequent jobs to check the status file before doing their real work. 这将允许后续作业在执行其实际工作之前检查状态文件。 Obviously, this isn't an ideal solution (too many points of failure, I suspect), but perhaps it's good enough for what you're trying to accomplish. 显然，这不是一个理想的解决方案（我怀疑故障点太多了），但是对于您要完成的任务而言，这也许已经足够了。