# How does the repetition of features across states at different time steps affect learning?

Let's say you are training a neural network in an RL setting, where the state (i.e. features/input data) can be the same for multiple successive steps (~typically around 8 steps) of an episode.

For example, an initial state might consist of the following values:

[30, 0.2, 0.5, 1, 0]


And then again the same state could be fed into the neural network for e.g. 6-7 times more, resulting in ultimately the following input arrays:

[[30, 0.2, 0.5, 1, 0],
[30, 0.2, 0.5, 1, 0],
...,
[30, 0.2, 0.5, 1, 0]]


I know that the value 0 in the feature set depicts that the weight for this feature results in insignificant value.

But what about the repetition of values? How does that affect learning, if it does at all? Any ideas?

The reason I did not provide this information in the first place, is because I thought there would be similarities in such cases across problems/domains of application. But it is also fine to make it more specific.

1. The output of the network is a probability among two paths. Our network has to select an optimal path based on some gathered network statistics.

2. I will be using A3C, as similar work in the bibliography has made progress.

3. The reason the agent is staying in the same state is the fact that the protocol can also make path selection decisions at the same time, without an actual update of network statistics. So in that case, you would have the same RTT for instance.

i. This is a product of concurrency in the protocol

ii. It is expected behavior

• You also asked this question here: datascience.stackexchange.com/q/74490/10640. Note that AI SE is probably the most appropriate site to ask questions related to reinforcement learning. Also, note that maybe nobody that visits our or other sites may know the answer to your question. I suggest you delete the other question from Data Science SE.
– nbro
May 20, 2020 at 12:09
• Just to clarify your question, so, for $k$ successive steps of an episode, the network that represents the value function or policy will always receive the same state as input? So, basically, the agent could be in the same state for multiple successive time steps? Is this your question? And are you asking how does this affect the training of the neural network?
– nbro
May 20, 2020 at 12:13
• @nbro Hi, thanks for replying. Yes I also asked the question there (with a link here ofc), after noticing that my previous question did not get any replies as well. I thought, and since DataScience seems to be a bigger community, that I could give it a shot. I also noticed that a lot of ML questions are asked on DataScience, so it kind of made sense. Regarding the question, yes you got it right, that is exactly what I am asking. May 20, 2020 at 12:20
• ML questions are on-topic on DS SE, but RL questions are more appropriate for AI SE. Anyway, what is the output of your network? How are you going to train it? Are you using or intend to use deep Q-learning? Maybe it may be useful to describe the actual problem that you want to solve (i.e. why would the agent stay in the same state for multiple steps)?
– nbro
May 20, 2020 at 12:42

In RL, neural networks may intuitively be thought of as using the input features as a representation that "identifies" the input state (or input state + action pair). Think back to the "tabular" RL setting that most people first study when they learn about RL. In tabular RL, you have a table of values (state values $$V(s)$$, or state-action values $$Q(s, a)$$), with unique entries in the table for every state. Such a table can perfectly identify states or, in other words, perfectly disambiguate different states.