4

The core differences between using $V(s)$ or $Q(s,a)$ are: $V(s)$ cannot be used stand-alone to decide a policy. You either need a separate policy function $\pi(a|s)$ that it is the value function for, or you can derive a policy from it if you have access to the environment's distribution model $p(r,s'|s,a)$ - the probability of receiving reward $r$ and ...


3

No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/unsupervised paradigms. Some RL algorithms don't even have a data-set as such. For instance, the vanilla tabular Q-learning does not use a data-set -- it will see ...


2

Residual Network are usually deeper and hence take more time to train. EfficientNet are trying to tackle this. However, the latest advice show that the architecture tend to play a crucial role in the performance of an RL algorithm, which might motivate you to do this. There is recent work on Neural Architecture Search applied to RL tasks (cf https://arxiv....


2

Thompson Sampling (TS) is used in the context of bandits, which is a special case of the RL problem. You can also use TS for the full RL problem, but that can lead to inefficient exploration. To know more about this issue, you could read the section 7.5 Reinforcement Learning in Markov Decision Processes (p. 62) of the tutorial A Tutorial on Thompson ...


2

Your setting (of randomly dropping out reward signals) impacts expected future reward by multiply everything by a common factor $(1-\epsilon)$. As reinforcement learning (RL) control is based on maximising expected future reward, and multiplying by a positive constant does not affect ranking of action values, all existing RL methods will cope just fine ...


2

This would mean we decrease the value of this state. Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy selects an action that is more likely in the target policy than in the behaviour policy. This over-estimate must happen, if your agent experiences some actions ...


1

Reinforcement learning already has the objective of maximising the sum of future expected reward. By making each reward the sum of all previous rewards, you will make the the difference between good and bad next choices low, relative to the overall reward guaranteed on each step. The best reward for the agent should be as direct measure of what you want it ...


1

Can't it be that the optimal policy thinks a state isn't that good and gives him a low value but perform best in comparison with other policies which have higher values for this state? No, this is not possible, and this is part of the definition of an optimal policy. You are asking if it is possible to construct a policy $\pi^?$ where for some state $s_z$, $...


1

A quick search about reinforcement learning applied to video games will lead you to countless tutorials that describe exactly what you're asking for. With images the way to go is usually deep reinforcement learning. A convolutional neural network (or any other deep learning architecture) is used to process the image and compress it to a latent vector used as ...


1

I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and experience replay), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN. Let's take a look at the observation method. def observation(self, observation): self....


1

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough. A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for choosing any action, where $\mathcal{A}$ is the set of all possible actions. I know how to code the $\epsilon$-greedy. Then you already know how to code the ...


1

Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of the curse of dimensions. However, whether one restricts this solution space by allowing only discrete values $[0.0, 0.1, 0.2, ..., 1.0]$ for the three actions ...


Only top voted, non community-wiki answers of a minimum length are eligible