How to clip output of CNN to specific list of colours

Question

Typical image GAN has output of shape (w,h,3) with values 0...1 that are then scaled to 0...255 to represent coloured image.

So in each w,h position we get 3 values that together represent single colour in RGB space.

I want to somehow limit that output to allow only specific colours as output. For example

colours = [[200, 20, 20], [20, 200, 20], [20, 20, 200], ...] 
number_of_allowed_colours = len(colours)

Ideally the allowed colours would be a part of input but given the complexity of task I am ok with starting first iterations of network with colours as part of network architecture.

So far I have unsuccessfully tried many approaches. The way I see it could be accomplished is having the last layer of network be of shape (w,h,number_of_allowed_colours) and then activation function is applied so that to map that 3rd dimension to corresponding colour triplet. One idea I am trying to expand is using something like tf.argmax in combination with tf.contrib.lookup.HashTable . Another option I am looking into is having the regular (w,h,3) shape layer but within activation round it up to closest match in colour. The problem here is the lack of gradient and the fact that there are no perfect algoritm to compare colours.

Edited: The loss function have to use the generated output image thus the solution has to fully differentiable. The output of network has to be (w,h,3) shaped image.

Answer 1

There's no way to know if this will actually work, but the suggestions below make your concepts differentiable.

(w,h,number_of_allowed_colours) and then activation function is applied so that to map that 3rd dimension to corresponding colour triplet. One idea I am trying to expand is using something like tf.argmax in combination with tf.contrib.lookup.HashTable . Another option I am looking into is having the regular (w,h,3)

I think you're really close here. You just need to make these operations differentiable.

For the first one, where you said:

activation function is applied ... tf.argmax ... lookup.HashTable .

What you want to do here looks a lot like an attention layer. Use activation="softmax" to get a weighted distribution over colors, where the weights sum to one. Take the weighted average of the colors.

Another option I am looking into is having the regular (w,h,3) but within activation round it up to closest match in colour. The problem here is the lack of gradient and the fact that there are no perfect algoritm to compare colours.

Again you've got the right idea, and are just missing a few details to make it differentiable. Here again I think you might just need a weighting that sums to one. You could get that by for each point, calculating the distance to each color-option and then taking softmax(-r**2) across the options. That way close options get a high weight, and distant options get weighted towards zero.

Averaging colors:

Don't forget gamma correction .