AttributeError: Tensor.op is meaningless when eager execution is enabled when using multiple feature inputs in Tensorflow federated

Question

I have a problem when inputting multiple feature inputs as follows:

feature_layer = tf.keras.layers.DenseFeatures(feature_cols)

feature_layer_inputs = {}
feature_layer_inputs['a'] = tf.keras.Input(shape=(1,), name='a', dtype=tf.int32)
feature_layer_inputs['b'] = tf.keras.Input(shape=(1,), name='b', dtype=tf.int32)

model = feature_layer(feature_layer_inputs)

for units in [64, 64]:
    model = tf.keras.layers.Dense(units, activation='relu')(model)
c_pred = tf.keras.layers.Dense(3, activation='softmax')(model) #classification 3 outputs

keras_model = tf.keras.Model(inputs=[v for v in feature_layer_inputs.values()], outputs=c_pred)

input_spec = collections.OrderedDict(
        x=collections.OrderedDict(
            a=tf.TensorSpec(shape=[None,], dtype=tf.int32),
            b=tf.TensorSpec(shape=[None,], dtype=tf.int32),
        y=tf.TensorSpec(shape=[None,], dtype=tf.int32))

def model_fn():
  # We _must_ create a new model here, and _not_ capture it from an external
  # scope. TFF will call this within different graph contexts.

  return tff.learning.from_keras_model(
      keras_model,
      #input_spec=preprocessed_sample_dataset.element_spec,
      input_spec=input_spec,
      loss=losses.SparseCategoricalCrossentropy(),
      metrics=[metrics.SparseCategoricalAccuracy()])


iterative_process = tff.learning.build_federated_averaging_process(
    model_fn,
    client_optimizer_fn=lambda: optimizers.Adam(learning_rate=client_lr),
    server_optimizer_fn=lambda: optimizers.SGD(learning_rate=server_lr))

state = iterative_process.initialize()

for round_num in range(1, NUM_ROUNDS+1):
    #state, tff_metrics = iterative_process.next(state, federated_train_data)
    state, tff_metrics = iterative_process.next(state, train_data)
    eval_model = keras_model
    eval_model.compile(optimizer=optimizers.Adam(learning_rate=client_lr),
                       loss=losses.SparseCategoricalCrossentropy(),
                       metrics=[metrics.SparseCategoricalAccuracy()])

    tff.learning.assign_weights_to_keras_model(eval_model, state.model)

    ev_result = eval_model.evaluate(x_test, y_test, verbose=0)

I get this code from this solution . However, I got the full traceback as follows. Eventually, I get an error 'AttributeError: Tensor.op is meaningless when eager execution'. It seems that there is something wrong with the built model especially the inputs inside the tf.keras.model function.

Traceback (most recent call last):
  File "accident_modify_uk_final3b.py", line 323, in <module>
    server_optimizer_fn=lambda: optimizers.SGD(learning_rate=server_lr))
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/federated_averaging.py", line 212, in build_federated_averaging_process
    stateful_delta_aggregate_fn, stateful_model_broadcast_fn)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/framework/optimizer_utils.py", line 360, in build_model_delta_optimizer_process
    @tff.tf_computation
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/api/computations.py", line 152, in tf_computation
    return computation_wrapper_instances.tensorflow_wrapper(*args)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 333, in __call__
    self._wrapper_fn)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 91, in _wrap
    concrete_fn = wrapper_fn(fn, parameter_type, unpack=None)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper_instances.py", line 52, in _tf_wrapper_fn
    target_fn, parameter_type, ctx_stack)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/impl/tensorflow_serialization.py", line 275, in serialize_py_fn_as_tf_computation
    result = target(*args)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/core/impl/utils/function_utils.py", line 455, in <lambda>
    return lambda: fn()  # pylint: disable=unnecessary-lambda
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/framework/optimizer_utils.py", line 364, in tf_init_fn
    stateful_model_broadcast_fn.initialize())
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/framework/optimizer_utils.py", line 227, in server_init
    _, optimizer_vars = _build_server_optimizer(model, optimizer)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/framework/optimizer_utils.py", line 123, in _build_server_optimizer
    apply_delta(delta=weights_delta)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 627, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
    *args, **kwds))
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)
AttributeError: in user code:

    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow_federated/python/learning/framework/optimizer_utils.py:112 apply_delta  *
        optimizer.apply_gradients(grads_and_vars, name='server_update')
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:508 apply_gradients  **
        "name": name,
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2420 merge_call
        return self._merge_call(merge_fn, args, kwargs)
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2427 _merge_call
        return merge_fn(self._strategy, *args, **kwargs)
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:590 _distributed_apply  **
        "update_" + var.op.name, skip_on_eager=True):
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:581 op
        return self._handle.op
    /home/anaconda3/envs/env1_TF2.1/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1113 op
        "Tensor.op is meaningless when eager execution is enabled.")

    AttributeError: Tensor.op is meaningless when eager execution is enabled.

How to solve this issue?

Answer 1

I believe the problem is that model_fn() is capturing keras_model from the global scope, rather than constructing a new instance when it is invoked.

From the documentation for tff.learning.build_federated_averaging_process :

model_fn : A no-arg function that returns a tff.learning.Model. This method must not capture TensorFlow tensors or variables and use them. The model must be constructed entirely from scratch on each invocation, returning the same pre-constructed model each call will result in an error.

There is a code comment in code above that alludes to this (copied here):

def model_fn():
  # We _must_ create a new model here, and _not_ capture it from an external
  # scope. TFF will call this within different graph contexts.

  return tff.learning.from_keras_model(
      keras_model,
      #input_spec=preprocessed_sample_dataset.element_spec,
      input_spec=input_spec,
      loss=losses.SparseCategoricalCrossentropy(),
      metrics=[metrics.SparseCategoricalAccuracy()])

To solve this, try calling tf.keras.Model constructor inside model_fn() . Likely the layers of the model will also need to be created inside as well, so it might be useful to have a create_keras_model() function that returns a new tf.keras.Model and call that helper from within model_fn() .