How to Identify Recurrent Connections in an Arbitrary Neural Network

Question

I am trying to implement Neuro-Evolution of Augmenting Topologies in C#. I am running into a problem with recurrent connections. I understand that, for a recurrent connectio

Answer 1

Okay, so instead of telling you to just not have recurrent connections, i'm actually going to tell you how to identify them.

First thing you need to know is that recurrent connections are calculated after all other connections and neurons. So which connection is recurrent and which is not depends on the order of calculation of your NN. Also, the first time when you put data into the system, we'll just assume that every connection is zero, otherwise some or all neurons can't be calculated.

Lets say we have this neural network: Neural Network

We devide this network into 3 layers (even though conceptually it has 4 layers):

Input Layer  [1, 2]
Hidden Layer [5, 6, 7]
Output Layer [3, 4]

First rule: All outputs from the output layer are recurrent connections.

Second rule: All outputs from the input layer may be calculated first.

We create two arrays. One containing the order of calculation of all neurons and connections and one containing all the (potentially) recurrent connections. Right now these arrays look somewhat like this:

Order of 
calculation: [1->5, 2->7 ]

Recurrent:   [ ]

Now we begin by looking at the output layer. Can we calculate Neuron 3? No? Because 6 is missing. Can we calculate 6? No? Because 5 is missing. And so on. It looks somewhat like this:

3, 6, 5, 7

The problem is that we are now stuck in a loop. So we introduce a temporary array storing all the neuron id's that we already visited:

[3, 6, 5, 7]

Now we ask: Can we calculate 7? No, because 6 is missing. But we already visited 6...

[3, 6, 5, 7,] <- 6

Third rule is: When you visit a neuron that has already been visited before, set the connection that you followed to this neuron as a recurrent connection. Now your arrays look like this:

Order of 
calculation: [1->5, 2->7 ]

Recurrent:   [6->7 ]

Now you finish the process and in the end join the order of calculation array with your recurrent array so, that the recurrent array follows after the other array. It looks somethat like this:

[1->5, 2->7, 7, 7->4, 7->5, 5, 5->6, 6, 6->3, 3, 4, 6->7]

Let's assume we have [x->y, y]

Where x->y is the calculation of x*weight(x->y)

And

Where y is the calculation of Sum(of inputs to y). So in this case Sum(x->y) or just x->y.

There are still some problems to solve here. For example: What if the only input of a neuron is a recurrent connection? But i guess you'll be able to solve this problem on your own...

Answer 2

I had a similar problem when i started implememting this paper. I don't know what your network looks like in the momen, so i'll explain to you what i did.

My network starts out as input & output layers only. To create connections and neurons i implemented some kind of DNA (in my case this is an array of instructions like 'connect neuron nr. 2 with neuron nr. 5 and set the weight to 0.4'). Every neuron in my network has a "layerNumber" which tells me where a neuron is in my network. This layerNumber is set for every in and output neuron. for inputneurons i used Double.minvalue and for outputneurons i used Double.maxvalue.

This is the basic setup. From now on just follow these rules when modifying the network:

• Whenever you want to create a connection, make sure the 'from' neuron has a layerNumber < Double.maxValue

• Whenever you want to create a connection, make sure that the 'to' neuron has a bigger layerNumber than the 'from' neuron.

• Whenever a connection is split up into 2 connections and a neuron between them, set the neurons layerNumber to NeuronFrom.layerNumber*0.5 + NeuronTo.layerNumber*0.5 This is important, you can't add them and simply divide by 2, because this would likely result in Double.maxValue + something, which would return some weird number (i guess an overflow would happen, so you would get a negative number?).

If you follow all the rules you should always have forwarding connections only. No recurrent ones. If you want recurrent connections you can create them by just swapping 'from' & 'to' while creating a new connection.

Pro tricks: Use only one ArrayList of Neurons. Make the DNA use ID's of neurons to find them, but create a 'Connection' class which will have the Neuron objects as attributes. When filtering your connections/neurons use ArrayList.stream().filter()

When later propagating trough the network you can just sort your neurons by layerNumber, set the inputValues and go trough all neurons using a for() loop. Just calculate the neurons outputvalue and transfer it to every neuron which has a connection where 'from' is == the current neuron.

Hope it's not too complicated...