How does the repetition of features across states at different time steps affect learning?

Question

Let's say you are training a neural network in an RL setting, where the state (i.e. features/input data) can be the same for multiple successive steps (~typically around 8 steps) of an episode.

For example, an initial state might consist of the following values:

[30, 0.2, 0.5, 1, 0]

And then again the same state could be fed into the neural network for e.g. 6-7 times more, resulting in ultimately the following input arrays:

[[30, 0.2, 0.5, 1, 0], 
 [30, 0.2, 0.5, 1, 0], 
 ..., 
 [30, 0.2, 0.5, 1, 0]]

I know that the value 0 in the feature set depicts that the weight for this feature results in insignificant value.

But what about the repetition of values? How does that affect learning, if it does at all? Any ideas?

Edit: I am going to provide more information as requested in the comments.

The reason I did not provide this information in the first place, is because I thought there would be similarities in such cases across problems/domains of application. But it is also fine to make it more specific.

1. The output of the network is a probability among two paths. Our network has to select an optimal path based on some gathered network statistics.

2. I will be using A3C, as similar work in the bibliography has made progress.

3. The reason the agent is staying in the same state is the fact that the protocol can also make path selection decisions at the same time, without an actual update of network statistics. So in that case, you would have the same RTT for instance.

   i. This is a product of concurrency in the protocol

   ii. It is expected behavior

Answer 1

In RL, neural networks may intuitively be thought of as using the input features as a representation that "identifies" the input state (or input state + action pair). Think back to the "tabular" RL setting that most people first study when they learn about RL. In tabular RL, you have a table of values (state values $V(s)$, or state-action values $Q(s, a)$), with unique entries in the table for every state. Such a table can perfectly identify states or, in other words, perfectly disambiguate different states.

In a non-tabular, function approximation setting, with function approximators such as Neural Networks, you can generally no longer uniquely identify every single state. Instead, you use approximate representations of these states, and the approximation implies that it's possible that you have multiple different states that look identical; they have identical input features. This is the case you're dealing with. Now, you specified explicitly that these multiple states with identical representations / input features follow each other up immediately in a single episode, but I don't think this detail is particularly important. You'd have exactly the same problems if these different states with identical representation showed up at different times within an episode. The only problem that you really have is a disambiguation problem: you don't know how to disambiguate these states, since they look identical to the network.

How significant that problem is depends on your domain. Based on your domain knowledge, do you expect the optimal action, or the optimal values, to be kind of similar in all these states that have identical features? If so, no problem! Your network already thinks they're the same anyway, so it will learn that the same actions / same values are the best in those states. But do you expect the optimal actions / true value functions to be wildly different in these states despite the fact that a network can't disambiguate them? In this case, the problem will be more severe because you can't realistically expect your network to learn the optimal actions / value functions for all these different states. At best, it can learn a weighted average among them (weighted by how commonly they occur in your training episodes).