9
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not ...
6
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then ...
5
Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.
The problem you describe is basically that you have situations where $Q^*(s, a_1) = Q^*(s, a_2) + \epsilon$ for some very small value $\epsilon$, where $a_1 \neq a_2$.
...
4
Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).
The Q function, however, measures the the value
of choosing a ...
4
More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
No, DQN (that is the commonly-used abbreviation by the way, there also is a dqn tag which you may wish to use) is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kinds of inputs.
The DQN algorithm ...
4
The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
However, if you want to explore tactics surrounding declaration, then I think the following applies:
There should be an initial round of actions where the agent may get to decide whether or ...
4
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.
I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more ...
3
Does it have to do with the reward function?
This seems likely to me. You have chosen a reward that is unusual in that it cross-links episodes. It is not really a reinforcement problem to optimise behaviour with respect to results of previous episode behaviour in this way. This might be an option for an evolutionary fitness context, if you have competing ...
3
120 inputs can be handled by a complex enough network. Dealing with high complexity is one of NN's strengths.
Using a (120,) vector or a (3,40) matrix is the same, they're still 120 inputs. Your binary encoding should work. Another option is a single (40,) vector, with 0 being "still in deck", 1 being "in hand", 2 being "on table", 3 being "already played". ...
3
Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).
I made the following changes to my code after closely examining the OpenAI baselines code:
Used PongFrameskip-v4 ...
3
Does the opponent's turn affect the calculated rewards?
Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.
As far as I know, the reward should only be the result of the agent's action right?
In a well-defined MDP, the reward should be a stochastic function of the current ...
3
The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.
If you're going to use someone else's ready-made code for your prioritized experience ...
3
Including exogenous variables in your state representation certainly can be useful, as long as you expect them to be relevant information for determining the action to pick. So, state features are not only useful if you expect your agent (through application of actions) to have (partial) influence on those state variables; you just want the state variables ...
3
The loss function for DQN algorithm is
\begin{equation}
L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2]
\end{equation}
Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ ...
2
I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward.
To me this implies that your profit per transaction is not the true reward function that you should be using. You don't ...
2
Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.
DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to ...
2
$ \gamma $ goes up to 1, but cannot be greater than or equal to 1 (this would make the discounted reward infinite).
The discount factor $ \gamma $ determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term ...
2
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were consecutives.
After redoing my experience selection method, my DQN is actually working and has reached about +200 points after 400000 experiences (about 500 episodes; ...
2
I have found some clues in Maei's thesis (2011): “Gradient Temporal-Difference Learning Algorithms.”
According to the thesis:
GTD2 is a method that minimizes the projected Bellman error (MSPBE).
GTD2 is convergent in non-linear function approximation case (and off-policy).
GTD2 converges to a TD-fixed point (same point as semi-gradient TD).
GTD2 is slower ...
2
Yes, you are exactly right. It is basically an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where ...
2
In Reinforcement Learning (RL), a reward function is part of the problem definition and should:
Be based primarily on the goals of the agent.
Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should ...
2
In my humble opinion, it seems like it is important to have them separated, if having a certain card can influence the result in some way that is not its prime value, instead of not only using the sum. But it depends on the game and its rules. For example:
If having 5 cards of hearts in the set of 15 cards makes you win the game, then if you only represent ...
2
Your problem is not that the environment is stochastic or dynamic. In fact you are using the terms slightly incorrectly. These terms do not usually refer to the fact that starting state can differ or goal locations can move episode-by-episode. They typically refer to behaviour of state transitions.
Although in your case you could view the initial state as ...
2
I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment.
Stochastic policy: Probabilistic(random) action choice for a given state.
Deterministic policy: one action is chosen for a given state.
Deterministic Policy Gradient algorithm ...
2
The initial state can vary during in both training and use, and how you decide to do this makes very little difference to Q-learning. The important factor is whether all state/action pairs relevant to optimal behaviour can be reached. As there is already randomness in any exploring policy, and in many environments as part of state transitions and reward ...
1
That may happen when the value of the state is bad. You can find the example and explain about that in the link below.
See this:https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682
1
The mistake is, that the Deep-Q-Network isn't able to return the needed action directly but it can make only a spatio-temporal prediction. In the sourcecode of the OP can be read in line 2:
action_values = self.parallel_model.predict(s_batch)
This line means that the output of the neural network is treated as an action which is feed into the controller. ...
1
You can't! As if you have a $\gamma$ greater than $1$ the specified sum for the q-learning will diverge! (goes to infinity in the future steps for $\gamma^n$). To know more about that please more scrutinize on the specified formula for the q-learning.
1
I suggest you to start reading this one on BERT and this one on GPT-2.
I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.
I'm not sure you want to create your chatbot using RL architecture at all. But if you want to implement such an idea the right way to go with that is by using iterative approach and ...
1
I assume OP understand basics of DQN - Time Difference (TD), target network and replay buffer. My experience is just my observations, it not necessarily generalize.
Some of those variants:
Double DQN: small difference in Time Difference propagation. Instead of propagating for maximum action of target network, Q taken form target network propagated for ...
Only top voted, non community-wiki answers of a minimum length are eligible
