9
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not ...
6
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this ...
5
To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can be found starting from the bottom of page 13.
Briefly, an action is described by picking a piece and then picking a move with it. The size of the board is 8 by ...
5
Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.
The problem you describe is basically that you have situations where $Q^*(s, a_1) = Q^*(s, a_2) + \epsilon$ for some very small value $\epsilon$, where $a_1 \neq a_2$.
...
5
More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
No, DQN (that is the commonly-used abbreviation by the way, there also is a dqn tag which you may wish to use) is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kinds of inputs.
The DQN algorithm ...
5
This answer will point the reader to potentially useful resources, but I can't ensure that the courses are good (because I have never followed them).
Free
Reinforcement Learning in the Open AI Gym (a small course that you can find in the YouTube channel suggested in the other answer) by Phil Tabor
The free course Advanced Deep Learning & Reinforcement ...
4
Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).
The Q function, however, measures the the value
of choosing a ...
4
The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
However, if you want to explore tactics surrounding declaration, then I think the following applies:
There should be an initial round of actions where the agent may get to decide whether or ...
4
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.
I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more ...
4
For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the uni. He covers basic algorithms like value iteration and policy iteration and also more advanced like deep q learning, covering all main python libraries (...
3
Does it have to do with the reward function?
This seems likely to me. You have chosen a reward that is unusual in that it cross-links episodes. It is not really a reinforcement problem to optimise behaviour with respect to results of previous episode behaviour in this way. This might be an option for an evolutionary fitness context, if you have competing ...
3
120 inputs can be handled by a complex enough network. Dealing with high complexity is one of NN's strengths.
Using a (120,) vector or a (3,40) matrix is the same, they're still 120 inputs. Your binary encoding should work. Another option is a single (40,) vector, with 0 being "still in deck", 1 being "in hand", 2 being "on table", 3 being "already played". ...
3
Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).
I made the following changes to my code after closely examining the OpenAI baselines code:
Used PongFrameskip-v4 ...
3
Does the opponent's turn affect the calculated rewards?
Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.
As far as I know, the reward should only be the result of the agent's action right?
In a well-defined MDP, the reward should be a stochastic function of the current ...
3
The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.
If you're going to use someone else's ready-made code for your prioritized experience ...
3
Including exogenous variables in your state representation certainly can be useful, as long as you expect them to be relevant information for determining the action to pick. So, state features are not only useful if you expect your agent (through application of actions) to have (partial) influence on those state variables; you just want the state variables ...
3
The loss function for DQN algorithm is
\begin{equation}
L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2]
\end{equation}
Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ ...
3
Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.
From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning
On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result ...
3
Whats does the target Q-values represent?
In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from that point on following a target policy. The target policy in Q learning is based on always taking the maximising action in each state, according to current ...
3
The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn't longer than 10-20 lines.
Here's a screenshot of the pseudocode of DQL (from the original paper) that highlights the Q target.
Here's the screenshot of Q-...
3
Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):
Neural network weight initialisation
Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, ...
3
I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?
That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is ...
3
The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.
It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target ...
3
MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.
Therefore, methods used in RL to solve MDPs will work to solve contextual bandits. You can either treat a contextual bandit as a series of 1-step episodes (with start state chosen randomly), or as a ...
3
I've implemented this exact scenario before; your approach would most likely be successful, but I think it could be simplified.
Therefore, when deciding on which action to pick, agent sets Q-values to 0 for all the illegal moves while normalizing the values of the rest.
In DQN, the Q-values are used to find the best action. To determine the best action in ...
3
You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items.
Provided both of these sets are finite, then you should still be able to approach this problem with DQN. ...
3
You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.
3
If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode.
You have two "...
3
Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?
No. That's not why Q-learning is model-free. Q-learning assumes that the underlying environment (FrozenLake or MountainCar, for example) can be modelled as a Markov decision ...
2
I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward.
To me this implies that your profit per transaction is not the true reward function that you should be using. You don't ...
Only top voted, non community-wiki answers of a minimum length are eligible
