[英]File “test_hdfs.py”, save_path = saver.save(sess, hdfs_path+“save_net.ckpt”) “Parent directory of {} doesn't exist, can't save.”.format(save_path))
How can I use the saver.save and FileWriter function to write checkpoint files and event logs into hdfs directly? 如何使用saver.save和FileWriter函数将检查点文件和事件日志直接写入hdfs?
I run my code: 我运行我的代码:
W = tf.Variable([[1,2,3],[3,4,5]], dtype=tf.float32, name='weights')
b = tf.Variable([[1,2,3]], dtype=tf.float32, name='biases')
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(init)
save_path = saver.save(sess, hdfs_path+"save_net.ckpt")
print("Save to path: ", hdfs_path)
When I replace the hdfs_path to a local path, it runs ok. 当我将hdfs_path替换为本地路径时,它运行正常。 But when I run a hdfs_path:
但是当我运行hdfs_path时:
File "test_hdfs.py", line 73, in <module>
save_path = saver.save(sess, hdfs_path+"save_net.ckpt")
File "/data/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1354, in save
"Parent directory of {} doesn't exist, can't save.".format(save_path))
This happens similarly when I use tf.summary.FileWriter function. 当我使用tf.summary.FileWriter函数时,也会类似地发生。 The program is stucked when I use hdfs_path.
当我使用hdfs_path时,程序卡住了。 When I use local_path, it runs ok.
当我使用local_path时,它运行正常。
My whole code is like this: 我的整个代码是这样的:
hdfs_path="hdfs://*"
local_path = "./"
with tf.Session(graph=tf.get_default_graph()) as sess:
W = tf.Variable([[1,2,3],[3,4,5]], dtype=tf.float32, name='weights')
b = tf.Variable([[1,2,3]], dtype=tf.float32, name='biases')
init = tf.group(tf.global_variables_initializer(),tf.local_variables_initializer())
saver = tf.train.Saver()
sess.run(init)
summary_writer = tf.summary.FileWriter(hdfs_path,graph_def=sess.graph_def)
saver.save(sess,save_path=hdfs_path+"save_net.ckpt")
When launching your TensorFlow program, the following environment variables must be set: 启动TensorFlow程序时,必须设置以下环境变量:
JAVA_HOME: The location of your Java installation. JAVA_HOME:Java安装的位置。 HADOOP_HDFS_HOME: The location of your HDFS installation.
HADOOP_HDFS_HOME:HDFS安装位置。 You can also set this environment variable by running:
您还可以通过运行以下命令来设置此环境变量:
shell source ${HADOOP_HOME}/libexec/hadoop-config.sh
LD_LIBRARY_PATH: To include the path to libjvm.so, and optionally the path to libhdfs.so if your Hadoop distribution does not install libhdfs.so in $HADOOP_HDFS_HOME/lib/native. LD_LIBRARY_PATH:如果Hadoop发行版未在$ HADOOP_HDFS_HOME / lib / native中安装libhdfs.so,则包括libjvm.so的路径,并可选地包括libhdfs.so的路径。 On Linux:
在Linux上:
shell export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${JAVA_HOME}/jre/lib/amd64/server:$HADOOP_HDFS_HOME/lib/native
The Hadoop jars must be added prior to running your TensorFlow program. 在运行TensorFlow程序之前,必须添加Hadoop jar。 The CLASSPATH set by ${HADOOP_HOME}/libexec/hadoop-config.sh is insufficient.
$ {HADOOP_HOME} /libexec/hadoop-config.sh设置的CLASSPATH不足。 Globs must be expanded as described in the libhdfs documentation:
必须按照libhdfs文档中的说明扩展Glob:
then uses shell find /hadoop_home/ -name *.jar|awk '{ printf("export CLASSPATH=%s:$CLASSPATH\\n", $0); }'
然后使用
shell find /hadoop_home/ -name *.jar|awk '{ printf("export CLASSPATH=%s:$CLASSPATH\\n", $0); }'
shell find /hadoop_home/ -name *.jar|awk '{ printf("export CLASSPATH=%s:$CLASSPATH\\n", $0); }'
to add hadoop jar to your path. shell find /hadoop_home/ -name *.jar|awk '{ printf("export CLASSPATH=%s:$CLASSPATH\\n", $0); }'
,将hadoop jar添加到您的路径。 After export all of the print, use shell python your_script.py
to run. 导出所有打印后,使用
shell python your_script.py
运行。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.