4
$\begingroup$

Let's say you are training a neural network in an RL setting, where the state (i.e. features/input data) can be the same for multiple successive steps (~typically around 8 steps) of an episode.

For example, an initial state might consist of the following values:

[30, 0.2, 0.5, 1, 0]

And then again the same state could be fed into the neural network for e.g. 6-7 times more, resulting in ultimately the following input arrays:

[[30, 0.2, 0.5, 1, 0], 
 [30, 0.2, 0.5, 1, 0], 
 ..., 
 [30, 0.2, 0.5, 1, 0]]

I know that the value 0 in the feature set depicts that the weight for this feature results in insignificant value.

But what about the repetition of values? How does that affect learning, if it does at all? Any ideas?

Edit: I am going to provide more information as requested in the comments.

The reason I did not provide this information in the first place, is because I thought there would be similarities in such cases across problems/domains of application. But it is also fine to make it more specific.

  1. The output of the network is a probability among two paths. Our network has to select an optimal path based on some gathered network statistics.

  2. I will be using A3C, as similar work in the bibliography has made progress.

  3. The reason the agent is staying in the same state is the fact that the protocol can also make path selection decisions at the same time, without an actual update of network statistics. So in that case, you would have the same RTT for instance.

    i. This is a product of concurrency in the protocol

    ii. It is expected behavior

$\endgroup$
  • $\begingroup$ You also asked this question here: datascience.stackexchange.com/q/74490/10640. Note that AI SE is probably the most appropriate site to ask questions related to reinforcement learning. Also, note that maybe nobody that visits our or other sites may know the answer to your question. I suggest you delete the other question from Data Science SE. $\endgroup$ – nbro May 20 at 12:09
  • $\begingroup$ Just to clarify your question, so, for $k$ successive steps of an episode, the network that represents the value function or policy will always receive the same state as input? So, basically, the agent could be in the same state for multiple successive time steps? Is this your question? And are you asking how does this affect the training of the neural network? $\endgroup$ – nbro May 20 at 12:13
  • $\begingroup$ @nbro Hi, thanks for replying. Yes I also asked the question there (with a link here ofc), after noticing that my previous question did not get any replies as well. I thought, and since DataScience seems to be a bigger community, that I could give it a shot. I also noticed that a lot of ML questions are asked on DataScience, so it kind of made sense. Regarding the question, yes you got it right, that is exactly what I am asking. $\endgroup$ – mkanakis May 20 at 12:20
  • 1
    $\begingroup$ ML questions are on-topic on DS SE, but RL questions are more appropriate for AI SE. Anyway, what is the output of your network? How are you going to train it? Are you using or intend to use deep Q-learning? Maybe it may be useful to describe the actual problem that you want to solve (i.e. why would the agent stay in the same state for multiple steps)? $\endgroup$ – nbro May 20 at 12:42
1
$\begingroup$

In RL, neural networks may intuitively be thought of as using the input features as a representation that "identifies" the input state (or input state + action pair). Think back to the "tabular" RL setting that most people first study when they learn about RL. In tabular RL, you have a table of values (state values $V(s)$, or state-action values $Q(s, a)$), with unique entries in the table for every state. Such a table can perfectly identify states or, in other words, perfectly disambiguate different states.

In a non-tabular, function approximation setting, with function approximators such as Neural Networks, you can generally no longer uniquely identify every single state. Instead, you use approximate representations of these states, and the approximation implies that it's possible that you have multiple different states that look identical; they have identical input features. This is the case you're dealing with. Now, you specified explicitly that these multiple states with identical representations / input features follow each other up immediately in a single episode, but I don't think this detail is particularly important. You'd have exactly the same problems if these different states with identical representation showed up at different times within an episode. The only problem that you really have is a disambiguation problem: you don't know how to disambiguate these states, since they look identical to the network.

How significant that problem is depends on your domain. Based on your domain knowledge, do you expect the optimal action, or the optimal values, to be kind of similar in all these states that have identical features? If so, no problem! Your network already thinks they're the same anyway, so it will learn that the same actions / same values are the best in those states. But do you expect the optimal actions / true value functions to be wildly different in these states despite the fact that a network can't disambiguate them? In this case, the problem will be more severe because you can't realistically expect your network to learn the optimal actions / value functions for all these different states. At best, it can learn a weighted average among them (weighted by how commonly they occur in your training episodes).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for you answer, very elaborate and I realize now that indeed I am facing a disambiguation problem. Although, I would like to add to this to make it more clear. Assuming repetitive states lead to optimal solutions, then there is no problem. How well will the NN be able to generalize if eventually a lot of the cases are repetitive? $\endgroup$ – mkanakis May 20 at 21:45
  • $\begingroup$ @mkanakis In fact, this phenomenon where multiple different states can look identical (or, in less extreme cases, just look similar by having similar-but-not-necessarily-identical inputs) is precisely where the power of NNs (and other function approximators) to generalise comes from. If every state looks equally different to every other state -- as is the case in tabular RL -- you have to learn separately for every state and cannot generalise. $\endgroup$ – Dennis Soemers May 21 at 8:39
  • $\begingroup$ Ah but if with "generalise" you're thinking of generalising to new states with different feature vectors that you didn't see before, then it can be a problem yes. If every single state you train on looks the same, your NN will "overfit" to states that look exactly like that. This is not a problem with repetitive states per se though, it's more a problem of your training data not being representative of all the data you're interested in. Such a mismatch can, in theory, also happen without repetitions in training data. $\endgroup$ – Dennis Soemers May 21 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.