Is there a tensorflow function that combines .h5 weights from different folds into 1 set of weights?

Question

I am training a model with k-fold validation and I am getting weights from each fold in the form of .h5 files. Is there a Tensorflow function that would allow me to combine all these weights into one .h5 file? So, it would take the weights for each layer and average them over the folds? Thanks in advance.

Answer 1

There is no TensorFlow function that can do what you are trying to. However, you can do it using other methods.

First, you will have to read each h5 file you have generated and locate where the data is stored in the hierarchy.

# A recursive function to get path of dataset element inside the 'encoded_weight.h5'

def traverse_datasets(hdf_file):

    def h5py_dataset_iterator(g, prefix=''):
        for key in g.keys():
            item = g[key]
            path = f'{prefix}/{key}'
            if isinstance(item, h5py.Dataset): # test for dataset
                yield (path, item)
            elif isinstance(item, h5py.Group): # test for group (go down)
                yield from h5py_dataset_iterator(item, path)

    for path, _ in h5py_dataset_iterator(hdf_file):
        yield path

import h5py
filename = "encoded_weight.h5"

hf = h5py.File(filename, "r")

for dset in traverse_datasets(hf):
    print('Path:', dset)
    print(hf[dset])
#     print(np.array(hf[dset]))   # Contains you array
    print('-----------------------')

Output:

Path: /dense/dense/bias:0
<HDF5 dataset "bias:0": shape (6,), type "<f4">
-----------------------
Path: /dense/dense/kernel:0
<HDF5 dataset "kernel:0": shape (13, 6), type "<f4">
-----------------------
Path: /dense_1/dense_1/bias:0
<HDF5 dataset "bias:0": shape (2,), type "<f4">
-----------------------
Path: /dense_1/dense_1/kernel:0
<HDF5 dataset "kernel:0": shape (6, 2), type "<f4">
-----------------------

The path is where your weights are actually stored. You can access it like.

w1 = tf.constant(hf['/dense/dense/bias:0'])
w1

Output:

<tf.Tensor: shape=(6,), dtype=float32, numpy=array([0., 0., 0., 0., 0., 0.], dtype=float32)>

You can do this for all layers.

And finally, compute the average:

(w1 + w2 ) / 2

Output:

<tf.Tensor: shape=(6,), dtype=float32, numpy=array([0.5, 0.5, 0.5, 0.5, 0.5, 0.5], dtype=float32)>

w2 is tensor of same shape but has elements 1.

After this, you can assign the weights using the set_weights property of a layer.

Take a look at this .

Note: If you think this will increase your accuracy you are wrong. Take a look at this