[英]Modifications in my tf.Graph change my NN weights initialization
Before starting, I am using python3.4.6 and tensorflow1.4.0 . 在开始之前,我正在使用python3.4.6和tensorflow1.4.0 。
I am aiming to compare an exact same NN architecture when training in tensorflow under different input pipelines. 我的目的是在不同输入管道下的张量流训练中,比较完全相同的NN体系结构。 For this purpose, I want to first ensure that same NN graph gets same initialization weights, regarding the input pipeline previously defined. 为此,对于先前定义的输入管道,我想首先确保相同的NN图获得相同的初始化权重。
I have seen that there exists two PRNGs in tensorflow, the graph one and the operations one. 我已经看到在张量流中存在两个PRNG,一个是图形,一个是操作。 I'm currently doing a tf.set_random_seed(777)
to set the graph's seed. 我目前正在做tf.set_random_seed(777)
来设置图的种子。
In order to make it easy, I post below a couple of codes that differ in the input pipeline and its weights output (a subset of them): 为了简单起见,我在下面发布了一些代码,它们在输入管道及其权重输出(它们的一部分)中有所不同:
Code 1 : 代码1 :
import tensorflow as tf
g = tf.Graph()
with g.as_default():
# Set graph seed
tf.set_random_seed(777)
# Input placeholders
input_x = tf.placeholder(tf.float32, shape=(None, 512, 512, 3))
labels = tf.placeholder(tf.int64, shape=(None, 1))
# Dummy placeholder
# dummy = tf.placeholder(tf.int32)
# Example Model
conv1_1 = tf.layers.conv2d(input_x, 8, 3, activation=tf.nn.relu, name='conv1_1')
conv1_2 = tf.layers.conv2d(conv1_1, 8, 3, activation=tf.nn.relu, name='conv1_2')
pool1 = tf.layers.max_pooling2d(conv1_2, 2, 2, name="pool1")
session_conf = tf.ConfigProto(log_device_placement=False)
with tf.Session(config=session_conf) as sess:
sess.run([tf.local_variables_initializer(), tf.global_variables_initializer()])
conv1_1_kernels = [v for v in tf.trainable_variables() if v.name == "conv1_1/kernel:0"][0]
print(sess.run(conv1_1_kernels)[:, :, 0, 0])
Output code 1: 输出代码1:
[[ 0.03720146 0.0177983 -0.18485998] [[0.03720146 0.0177983 -0.18485998]
[ 0.22072873 -0.14565685 0.21660429] [0.22072873 -0.14565685 0.21660429]
[-0.15442888 0.12140495 -0.05090818]] [-0.15442888 0.12140495 -0.05090818]
Code 2 : 代码2 :
import tensorflow as tf
g = tf.Graph()
with g.as_default():
# Set graph seed
tf.set_random_seed(777)
# Input placeholders
input_x = tf.placeholder(tf.float32, shape=(None, 512, 512, 3))
labels = tf.placeholder(tf.int64, shape=(None, 1))
# Dummy placeholder
dummy = tf.placeholder(tf.int32)
# Example Model
conv1_1 = tf.layers.conv2d(input_x, 8, 3, activation=tf.nn.relu, name='conv1_1')
conv1_2 = tf.layers.conv2d(conv1_1, 8, 3, activation=tf.nn.relu, name='conv1_2')
pool1 = tf.layers.max_pooling2d(conv1_2, 2, 2, name="pool1")
session_conf = tf.ConfigProto(log_device_placement=False)
with tf.Session(config=session_conf) as sess:
sess.run([tf.local_variables_initializer(), tf.global_variables_initializer()])
conv1_1_kernels = [v for v in tf.trainable_variables() if v.name == "conv1_1/kernel:0"][0]
print(sess.run(conv1_1_kernels)[:, :, 0, 0])
Output code 2: 输出代码2:
[[-0.20316723 0.01109874 -0.16709594] [[-0.20316723 0.01109874 -0.16709594]
[ 0.22850838 -0.10679846 -0.22449632] [0.22850838 -0.10679846 -0.22449632]
[-0.13468848 0.12664327 0.2225503 ]] [-0.13468848 0.12664327 0.2225503]]
These codes define my input pipeline through placeholders and define a simple NN graph. 这些代码通过占位符定义了我的输入管道,并定义了一个简单的NN图。 Then we start a tf.Session
as sess
to evaluate current weights on first channel of the first kernel in first layer conv1_1
. 然后,我们开始tf.Session
作为sess
以评估对在第一层中的第一内核的第一信道当前权conv1_1
。
The main difference between Code 1 and Code 2 is that we got the dummy
placeholder commented or not. 代码1和代码2之间的主要区别在于,我们是否对dummy
占位符进行了注释。 I have rerun both codes independently several times and weights are consistent . 我已经独立地重新运行了两个代码几次,并且权重是一致的 。
So, does anybody knows how to get same weights on my NN layers regarding if I define the dummy
placeholder or not? 那么,有没有人知道如何在我的NN层上获得关于是否定义dummy
占位符的相同权重? Why tensorflow PRNG depends on a previous placeholder (which does not need PRNG)? 为什么tensorflow PRNG依赖于先前的占位符(不需要PRNG)?
Any help will be appreciated! 任何帮助将不胜感激!
As it is said in the documentation : 正如文档中所说:
If the graph-level seed is set, but the operation seed is not: The system deterministically picks an operation seed in conjunction with the graph-level seed so that it gets a unique random sequence. 如果设置了图级别的种子,但未设置操作种子:系统确定性地选择一个操作种子和图级别的种子,以便获得唯一的随机序列。
Looks like this "deterministic pick" depends on graph contents. 看起来这个“确定性选择”取决于图形内容。
So to have reproducible results you also need to pass operation-level seed: 因此,要获得可重现的结果,您还需要传递操作级种子:
import tensorflow as tf
g = tf.Graph()
with g.as_default():
# Set graph seed
tf.set_random_seed(777)
# Input placeholders
input_x = tf.placeholder(tf.float32, shape=(None, 512, 512, 3))
labels = tf.placeholder(tf.int64, shape=(None, 1))
# Dummy placeholder
dummy = tf.placeholder(tf.int32)
# Example Model
conv1_1 = tf.layers.conv2d(
input_x, 8, 3,
activation=tf.nn.relu,
name='conv1_1',
kernel_initializer=tf.glorot_uniform_initializer(seed=1)
)
conv1_2 = tf.layers.conv2d(
conv1_1, 8, 3,
activation=tf.nn.relu,
name='conv1_2',
kernel_initializer=tf.glorot_uniform_initializer(seed=2)
)
pool1 = tf.layers.max_pooling2d(conv1_2, 2, 2, name="pool1")
session_conf = tf.ConfigProto(log_device_placement=False)
with tf.Session(config=session_conf) as sess:
sess.run([tf.local_variables_initializer(), tf.global_variables_initializer()])
conv1_1_kernels = [v for v in tf.trainable_variables() if v.name == "conv1_1/kernel:0"][0]
print(sess.run(conv1_1_kernels)[:, :, 0, 0])
I just added kernel_initializer
parameters. 我刚刚添加了kernel_initializer
参数。 They should be added to both scripts. 它们应该被添加到两个脚本中。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.