[英]Can I restart an iPython cluster from a notebook?
I'm just wondering if there is some python code or magics I can execute that will restart the ipython cluster.我只是想知道是否有一些我可以执行的 python 代码或魔法将重新启动 ipython 集群。 It seems like every time I change my code, it needs to be restarted.好像每次我更改我的代码时,它都需要重新启动。
A simple idea, seems too "hacky" for production: 一个简单的想法,似乎太生产了“hacky”:
Setup the Client
and define a simple function for testing. 设置Client
并定义一个简单的测试功能。
import ipyparallel as ipp
c = ipp.Client()
dv = c[:]
# simple function
@dv.remote(block=True)
def getpid():
import os
return os.getpid()
getpid()
[1994, 1995, 1998, 2001]
Define a function to restart the cluster. 定义重新启动集群的功能。 shutdown
with targets='all'
and hub=True
should kill the entire cluster. shutdown
with targets='all'
和hub=True
应该终止整个集群。 Then start a new cluster with !
然后开始一个新的集群!
or %sx magic command. 或%sx魔术命令。
import time
def restart_ipcluster(client):
client.shutdown(targets='all', hub=True)
time.sleep(5) # give the cluster a few seconds to shutdown
# include other args as necessary
!ipcluster start -n4 --daemonize
time.sleep(60) # give cluster ~min to start
return ipp.Client() # with keyword args as necessary
One drawback to this approach is that the DirectView needs to be re-assigned and any function decorated with dv.remote
or dv.parallel
needs to be re-executed. 这种方法的一个缺点是需要重新分配DirectView,并且需要重新执行用dv.remote
或dv.parallel
修饰的任何函数。
c = restart_ipcluster(c)
dv = c[:]
@dv.remote(block=True)
def getpid():
import os
return os.getpid()
getpid()
[3620, 3621, 3624, 3627]
Reading the source for ipyparallel Client
, the shutdown
method mentioned above has a keyword argument, restart=False
, but it's currently not implemented. 读取ipyparallel Client
的源代码,上面提到的shutdown
方法有一个关键字参数, restart=False
,但它当前没有实现。 Maybe the devs are working on a reliable method. 也许开发人员正在研究一种可靠的方法。
One can start/stop an IPython cluster using the class Cluster
.可以使用 class Cluster
启动/停止 IPython 集群。
Class representing an IPP cluster. Class 代表一个 IPP 集群。
ie one controller and one or more groups of engines.即一个 controller 和一组或多组引擎。
Can start/stop/monitor/poll cluster resources.可以启动/停止/监视/轮询集群资源。
All async methods can be called synchronously with a_sync
suffix, egcluster.start_cluster_sync()
所有异步方法都可以用_sync
后缀同步调用,例如cluster.start_cluster_sync()
Here's a demo:这是一个演示:
Start the cluster and check if it's running启动集群并检查它是否正在运行
import ipyparallel as ipp
cluster = ipp.Cluster(n=4)
# start cluster asyncronously (or cluster.start_cluster_sync() without await)
await cluster.start_cluster()
# Using existing profile dir: '/Users/X/.ipython/profile_default'
# Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
# <Cluster(cluster_id='1648978529-rzr8', profile='default', controller=<running>, engine_sets=['1648978530'])>
rc = ipp.Client(cluster_id='1648978529-rzr8')
cluster._is_running()
# True
This can be also seen in the Clusters tab of the Jupyter notebook:这也可以在 Jupyter notebook 的 Clusters 选项卡中看到:
Stop the cluster停止集群
# stop cluster syncronously
cluster.stop_cluster_sync()
cluster._is_running()
# False
Restart the cluster重启集群
# start cluster syncronously
cluster.start_cluster_sync()
cluster._is_running()
# True
Use cluster as a context manager使用集群作为上下文管理器
A cluster can be also used as a context manager using a with-statement ( https://ipyparallel.readthedocs.io/en/latest/examples/Cluster%20API.html#Cluster-as-a-context-manager ):集群也可以用作使用 with 语句的上下文管理器( https://ipyparallel.readthedocs.io/en/latest/examples/Cluster%20API.html#Cluster-as-a-context-manager ):
import os
with ipp.Cluster(n=4) as rc:
engine_pids = rc[:].apply_async(os.getpid).get_dict()
engine_pids
This way the cluster will only exist for the duration of the computation.这样集群只会在计算期间存在。
See also my other answer .另请参阅我的其他答案。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.