需要帮助理解 GPT-2s 源代码中的 Python 函数

Question

I'm walking through GPT-2s source code in Github. I'm trying to understand how it all works. I'm getting stumped on a function and I'm hoping somebody can explain to me what's happening.

https://github.com/nshepperd/gpt-2/blob/finetuning/src/model.py

The code can be found in model.py, in the link above. Here it is specifically:

def shape_list(x):
   """Deal with dynamic shape in tensorflow cleanly."""
   static = x.shape.as_list()
   dynamic = tf.shape(x)
   return [dynamic[i] if s is None else s for i, s in enumerate(static)]

I did some research on what Tensorflow.Shape() returns and on the differences between a static and dynamic shape here: How to understand static shape and dynamic shape in TensorFlow?

I also read through this series of articles: https://medium.com/analytics-vidhya/understanding-the-gpt-2-source-code-part-3-9796a5a5cc7c

Despite all that reading, I'm not entirely sure what's going. What isn't clear to me is the last statement:

return [dynamic[i] if s is None else s for i, s in enumerate(static)]

What exactly is it saying here? My guess is that the functions purpose is to determine if the value of X has been defined yet. If it hasn't then it will return the static shape, if it has then it will return the dynamic shape.

Am I way off here?

Answer 1

Your problem is not with anything Tensorflow, but with list comprehensions in python, which are a more pythonic way to define lists based on other iterables.

The last statement is (almost*) equivalent to:

ret = []
for i, s in enumerate(static):
  if s is None:
    ret.append(dynamic[i])
  else:
    ret.append(s)
return ret

* : About the "almost" above, the comprehension is actually more efficient, because internally it pre-allocates the memory for the whole result, while the loop appends s on every iteration, thus causing multiple allocations when extending the list, which is slower.