[英]Why do I get “_curses.error: cbreak() returned ERR” when using TensorFlow CLI Debugger?
I am trying to use the TensorFlow CLI debugger in order to identify the operation which is causing a NaN during training of a network, but when I try to run the code I get an error: 我正在尝试使用TensorFlow CLI调试器来识别在网络训练期间导致NaN的操作,但是当我尝试运行代码时出现错误:
_curses.error: cbreak() returned ERR
I'm running the code on an Ubuntu server, which I'm connecting to via SSH, and have tried to follow this tutorial . 我正在通过SSH连接到的Ubuntu服务器上运行代码,并尝试按照本教程进行操作 。
I have tried using tf.add_check_numerics_ops()
, but the layers in the network include while loops so are not compatible. 我尝试使用
tf.add_check_numerics_ops()
,但是网络中的层包括while循环,因此不兼容。 This is the section of code where the error is being raised: 这是发生错误的代码部分:
import tensorflow as tf
from tensorflow.python import debug as tf_debug
...
#Prepare data
train_data, val_data, test_data = dataset.prepare_datasets(model_config)
sess = tf.Session()
sess = tf_debug.LocalCLIDebugWrapperSession(sess)
# Create iterators
handle = tf.placeholder(tf.string, shape=[])
iterator = tf.data.Iterator.from_string_handle(handle, train_data.output_types, train_data.output_shapes)
mixed_spec, voice_spec, mixed_audio, voice_audio = iterator.get_next()
training_iterator = train_data.make_initializable_iterator()
validation_iterator = val_data.make_initializable_iterator()
testing_iterator = test_data.make_initializable_iterator()
training_handle = sess.run(training_iterator.string_handle())
...
and the full error is: 而完整的错误是:
Traceback (most recent call last):
File "main.py", line 64, in <module>
@ex.automain
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 137, in automain
self.run_commandline()
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 260, in run_commandline
return self.run(cmd_name, config_updates, named_configs, {}, args)
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/experiment.py", line 209, in run
run()
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/run.py", line 221, in __call__
self.result = self.main_function(*args)
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/sacred/config/captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "main.py", line 95, in do_experiment
training_handle = sess.run(training_iterator.string_handle())
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/framework.py", line 455, in run
is_callable_runner=bool(callable_runner)))
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/local_cli_wrapper.py", line 255, in on_run_start
self._run_start_response = self._launch_cli()
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/wrappers/local_cli_wrapper.py", line 431, in _launch_cli
title_color=self._title_color)
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/cli/curses_ui.py", line 492, in run_ui
self._screen_launch(enable_mouse_on_start=enable_mouse_on_start)
File "/home/enterprise.internal.city.ac.uk/acvn728/.local/lib/python3.5/site-packages/tensorflow/python/debug/cli/curses_ui.py", line 445, in _screen_launch
curses.cbreak()
_curses.error: cbreak() returned ERR
I'm pretty new to using Ubuntu (and TensorFlow), but as far as I can tell the server does have ncurses installed, which should allow the required curses based interface: 我对使用Ubuntu(和TensorFlow)很陌生,但是据我所知服务器确实已安装ncurses,这应该允许基于curses的必需接口:
acvn728@america:~/MScFinalProject$ dpkg -l '*ncurses*' | grep '^ii'
ii libncurses5:amd64 6.0+20160213-1ubuntu1 amd64 shared libraries for terminal handling
ii libncursesw5:amd64 6.0+20160213-1ubuntu1 amd64 shared libraries for terminal handling (wide character support)
ii ncurses-base 6.0+20160213-1ubuntu1 all basic terminal type definitions
ii ncurses-bin 6.0+20160213-1ubuntu1 amd64 terminal-related programs and man pages
ii ncurses-term 6.0+20160213-1ubuntu1 all additional terminal type definitions
Problem solved! 问题解决了! The solution was to change
解决的办法是改变
sess = tf_debug.LocalCLIDebugWrapperSession(sess)
to 至
sess = tf_debug.LocalCLIDebugWrapperSession(sess, ui_type="readline")
This is similar to the solution to this question , but II think it is important to note that they are different because a) it refers to a different function and a different API and b) I wasn't trying to run from an IDE, as mentioned in that solution. 这类似于该问题的解决方案,但是II认为必须注意它们是不同的,因为a)所指的是不同的函数和不同的API,并且b)我不是要从IDE运行,因为在该解决方案中提到。
cbreak
would return ERR
if you run a curses application that is not on a real terminal (ie, something that works with POSIX termios calls ). 如果您运行不在真实终端上的curses应用程序(即与POSIX termios调用兼容的应用程序),则
cbreak
将返回ERR
。
From the description, 根据描述,
but the layers in the network include while loops so are not compatible
但是网络中的层包括while循环,因此不兼容
it does not seem you are running in a terminal. 看来您没有在终端上运行。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.