All Questions
843 questions
1
vote
1
answer
72
views
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
- 187
0
votes
1
answer
406
views
Which policy do I need to use in updating Q function?
Policy function can be of two types: deterministic policy and stochastic policy.
Deterministic policy is of the form $\pi : S \rightarrow A$
Stochastic policy is defined using conditional probability ...
- 187
1
vote
1
answer
96
views
In addition to the reward function, which other functions do I need to implement Q-learning?
In general, $Q$ function is defined as
$$Q : S \times A \rightarrow \mathbb{R}$$
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$
$\alpha$ and $\gamma$...
- 187
1
vote
2
answers
2k
views
What is the difference between a reward and a value for a given state?
I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
- 57
1
vote
0
answers
64
views
What is the conceptual difference between convolutional neural networks and auto-encoders?
I'm familiar with Auto-Encoders and I'm about to dive into CNNs. By having a look at the most important component of a CNN, the filter:
I wonder how it is different from Auto-Encoders:
For me, it ...
- 205
1
vote
0
answers
50
views
Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?
I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers
Learning rates for Q-learning, by Eyal Even-Dar et al.;
...
4
votes
1
answer
829
views
Is the Bandit Problem an MDP?
I've read Sutton and Barto's introductory RL book. They define a policy as a mapping from states to probabilities of selecting each possible action. If the agent is following policy $\pi$ at time $t$, ...
- 225
0
votes
2
answers
829
views
What is the exact difference between distributional semantics and distributed semantics?
While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein
Distributional ...
- 3,990
3
votes
1
answer
5k
views
What is the difference between terminal state, nonterminal states and normal states?
In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following:
Each episode ends in a special state called the terminal state
But the ...
- 573
1
vote
2
answers
7k
views
What is the difference between "ground truth" and "ground-truth labels"?
I'm aware that the ground-truth of the example at the top left-hand corner of the image below is "zero"
However, I am confused about the meaning of the terms ground truth and ground-truth ...
- 217
2
votes
2
answers
192
views
How is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter?
From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on ...
- 221
2
votes
1
answer
253
views
Is categorical encoding a type of word embedding?
Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding.
I can see ...
- 3,990
8
votes
1
answer
4k
views
Deep Q-Learning "catastrophic drop" reasons?
I am implementing some "classical" papers in Model Free RL like DQN, Double DQN, and Double DQN with Prioritized Replay.
Through the various models im running on ...
- 81
4
votes
1
answer
2k
views
Where do the feature extraction and representation learning differ?
Feature selection is a process of selecting a subset of features that contribute the most.
Feature extraction allows getting new features that are not actually present in the given set of features.
...
- 3,990
1
vote
0
answers
127
views
What is the difference between ERL and EA by considering it as RL?
I am currently studying as an MSCS student and my research is based on Evolutionary Algorithm as Reinforcement Learning, and I am confused about the following terms:
What is the difference between ...
- 11
-1
votes
0
answers
62
views
CartPoleV0 model is not getting trained in even after 1500+ episodes using deep Q-learning
I am new to deep Q learning and trying to train the open AI cartpole_V0 game using deep Q learning. Here is my code:
...
- 99
0
votes
2
answers
1k
views
What is the difference between the "equal error rate" and "detection cost function" metrics?
I was designing a multi-speaker identification model, so I searched for some metrics that one may use. I found two metrics:
EER (equal error rate)
DCF (detection cost function)
What is the ...
- 361
2
votes
0
answers
457
views
Update Rule with Deep Q-Learning (DQN) for 2-player games
I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
- 31
0
votes
1
answer
419
views
Is a genetic algorithm efficient for a snake game?
I am working on a DIY project in which I want to be able to train a neural network to play Snake.
Is a genetic algorithm an efficient way of training a network for this application?
For a GA, what ...
- 1
6
votes
2
answers
8k
views
What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?
I have studied in the past different algorithms, i.e. DQN, DDQN, REINFORCE, A3C, PPO, TRPO, so on. I am doing an internship this summer where I have to use a multi-armed bandit (MAB). I am a bit ...
0
votes
0
answers
1k
views
Reward firstly increase, but after more episodes, start decrease, and weights diverges
I'm making a simple deep Q learning algorithm, with cartpole-v1 env.
Like you can see in chart, after many episodes the reward decrease, some possible reasons?
The exploration vs axplotation algorithm ...
1
vote
2
answers
279
views
What is the difference between applying shallow-learning methods repeatedly and deep learning?
In the book Deep Learning with Python, François Chollet writes (section 1.2.6, page 18)
In practice, there are fast-diminishing returns to successive applications of shallow-learning methods, because ...
- 13
-1
votes
0
answers
83
views
Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?
I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
- 49
4
votes
1
answer
156
views
In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
1
vote
1
answer
733
views
How to scale all positive continuous reward?
My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
0
votes
0
answers
208
views
Comparing heuristics in A* search and rescue operation
I was reading a research paper titled A Comparative Study of A-star Algorithms for Search and rescue in Perfect Maze (2011).
I have some doubts regarding it:
1.
The Evaluation Function of $\mathrm{A}^...
14
votes
1
answer
11k
views
What are the consequences of layer norm vs batch norm?
I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the ...
- 1,369
1
vote
1
answer
396
views
How to properly resume training of deep Q-learning network?
I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
- 13
0
votes
1
answer
166
views
Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning
I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
- 393
0
votes
1
answer
162
views
Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?
Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete.
Is ...
- 1
4
votes
1
answer
211
views
What is the relation between self-taught learning and transfer learning?
I am new to transfer learning and I start by reading A Survey on Transfer Learning, and it stated the following:
according to different situations of labeled and unlabeled data in the source domain, ...
- 361
2
votes
0
answers
162
views
MicroPython MicroMLP: How do I reward the program based on state?
I have been trying to use MicroMLP to teach a small neural network to converge to correct results. Ultimately, I want to have three outputs, one which is high priority (...
- 121
4
votes
1
answer
590
views
What is the difference between a distribution model and a sampling model in Reinforcement Learning?
The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as
something that mimics the behavior of the environment, or more generally, that ...
- 141
2
votes
2
answers
2k
views
Reinforcement Learning for an environment that is non-markovian [closed]
I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can ...
- 123
3
votes
4
answers
315
views
Is there a relationship between Computer Algebra and NLP?
My intuition is that there is some overlap between understanding language and symbolic mathematics (e.g. algebra). The rules of algebra are somewhat like grammar, and the step-by-step arguments get ...
- 33
2
votes
1
answer
2k
views
How to avoid being stuck local optima in q-learning and q-network
When using the Bellman equation to update q-table or train q-network to fit greedy max values, the q-values very often get to the local optima and get stuck although randomization rate ($\epsilon$) ...
- 1,318
1
vote
1
answer
425
views
Q-learning in gridworld with random board
I'm trying to use Q-learning in order to solve Wumpus world environment.
Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
- 25
1
vote
1
answer
347
views
Compute state space from variables in Q-learning (RL)
I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state.
Let's say, in my problem, there are the following variables, which I'm using to compute state:
...
- 25
5
votes
2
answers
8k
views
What is the difference between a language model and a word embedding?
I am self-studying applications of deep learning on the NLP and machine translation.
I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".
...
- 373
2
votes
0
answers
96
views
Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?
For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
- 75
2
votes
1
answer
1k
views
What is a good convergence criterion for Q-learning in a stochastic environment?
I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
- 75
5
votes
1
answer
1k
views
How would I compute the optimal state-action value for a certain state and action?
I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:
Learning rate = 0.1
Discount factor = 0.95
...
- 215
5
votes
1
answer
1k
views
Why does Q-learning converge under 100% exploration rate?
I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
- 215
1
vote
2
answers
2k
views
Is there any situation in which breadth-first search is preferable over A*?
Is there any situation in which breadth-first search is preferable over A*?
- 11
3
votes
1
answer
145
views
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
Knowing that, my main question concerns Q-learning and off-policy methods:
For me, there is something weird in ...
- 165
2
votes
2
answers
673
views
What is the target output for updating a Deep Q Network
I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
- 197
0
votes
1
answer
426
views
What are the popular approaches to estimating the Q-function?
I need the q-value for my RL training, there are some approaches:
Brute-force the action sequence (this won't work for long sequence)
Use a classic algorithm to optimise and estimate (this ain't much ...
- 1,318
5
votes
3
answers
884
views
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
- 197
3
votes
1
answer
1k
views
What is a "learned policy" in Q-learning?
I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
user43972
5
votes
1
answer
6k
views
What is the difference between out of distribution detection and anomaly detection?
I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural ...
- 243