I'm working on an implementation of NEAT, which evolves neural networks with small and sparse topologies.

Evaluating a sparse and possibly recurrent network requires a different approach than the matrix operations of dense networks, and I'm trying to wrap my head around the order at which nodes should be evaluated.

I've set up a simple example:

• Every node (except inputs) starts at 0. (t0)
• Every weight is 1. There's no activation function neither biases.

Assuming that the algorithm has the same intuition we humans do: evaluate the nodes connected to the input, then the "aggregation" node, then the output. How should it decide whether to evaluate node [3] before node [4] or vice-versa?

By starting (t0) at [3], the value of [4] is 0. And vice-versa.

The side-effect of this behavior appears when comparing two networks. Say you have the network above, and a copy of it where [3] and [4] are inverted. I feel like both should return the same result for the same input in order to evolution work properly.

Any thoughts?