keras / tensorflow中的树结构输入

Question

For some school project I am trying to implement a tree convolution as described in "Convolutional Neural Networks over Tree Structures for Programming Language Processing" Lili Mou, et al.

Goal

Basically, the outcome should be a neural network. The samples to this network are binary trees whose nodes have a fixed length features such as 1xN . The challenging part for me has been the freedom in the tree shape. This means a sample tree may have any number of nodes in any shape. A left-deep tree, right-deep tree, complete tree are all possible. The only constraint is they all should be binary trees.

The tree convolution on a sample tree is defined with 3 weight matrices W_p, W_l, W_r . These weights are used for each node in the tree to generate another tree of the same shape but with different features such as 1xM if the weights are of shape NxM . For each node its feature gets multiplied by W_p and its children by W_l, W_r so the node in the new tree will contain information about itself and both its children.

Then there comes finally a dynamic pooling layer over all the tree nodes to have a 1xM flattened vector in the end so that it could be fed into a Dense Layer for example. The way it works is they call each entry of 1xM vectors a channel. Then for each channel the maximum value over all nodes is returned to have a 1xM vector.

Problem

This was a quick explanation of the paper. Now the problem as I said in the first paragraph is the varying number of children of these binary trees. First I tried to use Keras, but obviously it needs fixed-size input for Layers. Then it occured to me I can use array implementation of binary trees to encode each tree in a fixed-size fashion. This means for example a parent at node i would have its children at 2*i and 2*i+1 . Whenever there are not children in some places, put N zeros for padding if the features are of length N .

This required me to have information about the maximum index over all trees such that I can create some AxN array where A is the maximum indexing used in this fixed-size schema. Sadly, the input trees may be really deep with fewer nodes so to encode 16 nodes I have to create a 60000xN or 6000xN array most of which gets zero padded just because the tree is not well-balanced.

Then I switched to a custom SGD implementation where I defined Dense, Tree Convolution, Dynamic Pooling quickly. The forward pass was really easy. In the backprop, however, I got it to the point I can propagate derivatives from Dense to the Pooling to the tree before the pooling and do a weight update in that tree, but not for the before trees. Since Keras/TF handles differentiation in the background it was easier indeed.

Now I feel really stuck between choosing approaches for this problem. Obviously Keras/TF has lots of functionality available for designing such a network. Should there be an efficient way of passing this tree structured data to these libraries so for 30 nodes I do not end up creating 60000 nodes with 59970 zero vectors? The idea of generating 6000 or 60000 nodes for some 15 nodes is just crazy at this point even if you got the best GPU out there.

Or should I work on deriving the derivative equations on the paper to continue the custom SGD implementation?

For reference, this was how it looked like with Keras, with the inefficient encoding of the trees I mentioned above.

class MyLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create a trainable weight variable for this layer
        self.kernel = self.add_weight(name='kernel',
                                      shape=(3, input_shape[2], self.output_dim[1]),
                                      initializer='ones',
                                      trainable=True)
        super(MyLayer, self).build(input_shape)  # Be sure to call this at the end

    def call(self, x):
        _, tree_size, feature_size = K.int_shape(x)

        new_tree = []
        for i in range(tree_size // 2):
            parent = tf.gather_nd(x, (0,i))
            left = tf.gather_nd(x, (0, 2*i + 1) )
            right = tf.gather_nd(x, (0, 2*i + 2))
            p_l_r = K.expand_dims(K.stack([parent, left, right]), axis = 1)
            product = K.sum(K.batch_dot(p_l_r, self.kernel), axis = 0)
            new_tree.append(product)
        for j in range (tree_size //2, tree_size):
            parent = tf.gather_nd(x, (0, j))
            parent = K.expand_dims(parent, axis = 0)
            product = K.dot(parent, self.kernel[0])
            new_tree.append(product)

        new_tree = K.stack(new_tree, axis = 1)
        return new_tree
    def compute_output_shape(self, input_shape):
        return (input_shape[0], self.output_dim[0], self.output_dim[1])

Answer 1

Tensorflow used to have a decision tree implementation. You can see the data structures (variables) it used here: https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/tensorflow/contrib/tensor_forest/python/tensor_forest.py#L155

It shows that you can implement a tree by creating a 2D tensor of shape (max_nodes, max_children) . The (i, j) entry has an integer telling the index of the jth child of the ith node in the same tensor. So an upside-down-v shaped binary tree with three nodes would be [[1, 2], [-1, -1], [-1, -1]] .

You could easily create a second tensor to hold the features, where the ith row hold the features for the ith node. Then it would be possible to perform the convolution operation you mentioned, although it would require looping. I don't see a way to vectorize it, but that's the cost of using a (somewhat) sparse representation.