9
$\begingroup$

I'm attempting to program my own system to run a neural network. To reduce the number of nodes needed, it was suggested to make it treat rotations of the input equally.

My network aims to learn and predict Conway's Game of Life by looking at every square and its surrounding squares in a grid, and giving the output for that square. Its input is a string of 9 bits:

Glider

The above is represented as 010 001 111.

There are three other rotations of this shape however, and all of them produce the same output:

Glider rotations

My network topology is 9 input nodes and 1 output node for the next state of the centre square in the input. How can I construct the hidden layer(s) so that they take each of these rotations as the same, cutting the number of possible inputs down to a quarter of the original?

Edit:

There is also a flip of each rotation which produces an identical result. Incorporating these will cut my inputs by 1/8th. With the glider, my aim is for all of these inputs to be treated exactly the same. Will this have to be done with pre-processing, or can I incorporate it into the network?

$\endgroup$
  • $\begingroup$ Outstanding question! I have a similar, impeding issue with my own project and will be very interested to learn about the most efficient techniques to reduce for symmetry. $\endgroup$ – DukeZhou Aug 7 '17 at 18:25
  • $\begingroup$ @DukeZhou I get the impression that it will take a while to get an answer however. I'm prepared to use up my meagre supply of rep to set up a bounty if needed... $\endgroup$ – Aric Aug 7 '17 at 18:38
  • $\begingroup$ Another solution is to pre-process the input, such that all 4 rotations are converted into the same image before being fed to the network. $\endgroup$ – BlueMoon93 Aug 11 '17 at 8:01
4
$\begingroup$

If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.

You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.

You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:

from keras.models import Sequential
from keras.layers import Input, Dense
from keras.models import Model
from keras import optimizers
from keras.utils.np_utils import to_categorical
import helper

x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
y_binary = to_categorical(y_)

model = Sequential()
model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
model.add(Dense(2, activation='softmax'))
adam=optimizers.Adam()
model.compile(optimizer=adam,
              loss='categorical_crossentropy',
              metrics=['acc'])
model.fit(x_, y_binary, epochs=100)
$\endgroup$
  • $\begingroup$ The network I will be using will be simulated by a class written by myself, hence the memory concerns. $\endgroup$ – Aric Aug 8 '17 at 12:02
  • $\begingroup$ If you want reduce memory usage by the network then reducing the number of possible inputs (by rotating) will help to have a smaller network. The easier the learning task is the smaller network needed. In this case pre-processing would be better. Nevertheless NN is for concept learning and to learn the concept of Game of Life you should feed in all patterns. If your aim is to absolutely minimize memory footprint solve the problem on a linear way. $\endgroup$ – Manngo Aug 8 '17 at 12:27
  • $\begingroup$ If memory is not a concern, I would rather the network performs this operation for the same reasons you stated. Pre-processing removes some of the task from the network, simplifying it. $\endgroup$ – Aric Aug 8 '17 at 12:31
  • $\begingroup$ Correct, then go for the pre-processing. I think with this the question is answered. You may chose to implement it if you run into issues with memory. Tip: use float for the weights that takes only 32 bits instead of double that takes 64. This will use less memory. $\endgroup$ – Manngo Aug 8 '17 at 12:56
4
$\begingroup$

You have identified an optimization in your problem space and desire to bake this into your neural net. I suggest preprocessing: Compose your optimization with a neural net that does a subset of what you want.

In other words, normalize your input by manually coding a rotation algorithm that rotates inputs to capture the equivalence highlighted in your post. Then feed the output of this transformation to your neural net, for training and all other uses. This means you are training the neural net to tackle the sub-problem you identified - rotations are redundant.

Test your normalizer by generating random input, rotating it to all four potential transformations, run the normalizer on each one, then check that they are all equivalent.

$\endgroup$
1
$\begingroup$

To be purist about it, begin with considering the input differently, as a circular array of size four, each item containing a pair of bits, and additionally a center bit:

... 01, 01, 11, 10 ...

0

Throughout the design of the network, continue this circular structure and center point paradigm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.