I'm working on an implementation of NEAT, which evolves neural networks with small and sparse topologies.

Evaluating a sparse and possibly recurrent network requires a different approach than the matrix operations of dense networks, and I'm trying to wrap my head around the order in which nodes should be evaluated.

I've set up a simple example:

 - Every node (except inputs) starts at 0. (t0)
 - Every weight is 1. There's no activation function or biases.

[![enter image description here][1]][1]

Assuming that the algorithm has the same intuition we humans do: evaluate the nodes connected to the input, then the "aggregation" node, then the output.
How should it decide whether to evaluate node [3] before node [4] or vice-versa?

By starting (t0) at [3], the value of [4] is 0. And vice-versa.

The side-effect of this behavior appears when comparing two networks. Say you have the network above, and a copy of it where [3] and [4] are inverted. I feel like both should return the same result for the same input in order for evolution to work properly.

Any thoughts?

  [1]: https://i.sstatic.net/Vtye3.png