All Questions
843 questions
2
votes
0
answers
224
views
Reduction of state space of the game Connect Four to apply RL algorithms SARSA and Q-Learning
I would like to implement the reinforcement learning algorithms SARSA and Q-Learning for the board game Connect Four.
I am familiar with the algorithms and know about their limitations regarding large ...
- 21
1
vote
0
answers
97
views
Can Q-learning be used for my scenario, and how might I do so?
I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help.
Scenario: I have a ...
0
votes
1
answer
299
views
Alternatives to neural networks for function approximation in Q learning?
I want to know if there is anything other than neural networks (or Deep NNs) that I can effectively use to perform function approximation? I am asking this w.r.t to the use of approximators in Q ...
6
votes
1
answer
2k
views
When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when ...
0
votes
0
answers
175
views
What do we actually 'approximate' when dealing with large state spaces in Q-learning?
I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has ...
0
votes
0
answers
117
views
How to deal with Q-learning having low variance in predicted Q-values?
I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning.
I've ...
- 454
2
votes
2
answers
208
views
How will MLOps and lifelong learning be complementary?
According to [1], in MLOps, continuous training is
a new property, unique to ML systems, that's concerned with automatically retraining and serving the models.
While lifelong/incremental learning ...
- 967
2
votes
1
answer
813
views
Why would SARSA diverge (but not Expected SARSA or Q-learning)?
In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
- 121
2
votes
0
answers
153
views
What is the difference between Probabilistic Graphical models and Graph Neural networks?
While going over PGMs and GNNs, it seems like both leverage the graph data structure. The former has been used to represent causal associations (among other things), while the latter has a varied set ...
- 670
1
vote
0
answers
61
views
Multi-armed Bandit in optimization on graph edges selection
I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it.
I am working on computer networking optimization.
In the simplest ...
- 21
1
vote
1
answer
661
views
How is the VAE related to the Autoencoding Variational Bayes (AEVB) algorithm?
I am familiar with the variational autoencoder, but not totally clear on what exactly the AEVB is.
In the original VAE paper (by Kingma and Welling), he uses both the terms variational autoencoder and ...
- 243
0
votes
1
answer
128
views
In this example of fuzzy c-means, what is the difference between "sigma" and "center" for the clusters?
In this example, what exactly do "Cluster" and "Sigma" mean? (They chose random coordinates for the three centroids of the groups)
Centers: Cluster centers, returned as a ...
- 121
1
vote
0
answers
25
views
Transferring a Q-learning policy to larger instances
How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances?
Some more details on the problem:
I am currently trying to derive a decision policy for ...
1
vote
1
answer
338
views
What is the difference between "Syllogism" and "Law of Syllogism"?
The logical arguments are the basis for Artificial Intelligence. That is why I picked AI community to ask my question.
Reading from Wikipedia,
A syllogism is a kind of logical argument that applies ...
- 13
2
votes
1
answer
173
views
Closed networks vs Networks with a removed delay to predict new data
I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data.
From Matlab's app generated scripts we see:
% ...
6
votes
2
answers
8k
views
When to use Value Iteration vs. Policy Iteration
Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
- 541
2
votes
1
answer
1k
views
How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?
I am trying to use reinforcement learning to solve a task and compare its performance to humans.
The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
- 223
1
vote
1
answer
312
views
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{...
- 111
2
votes
1
answer
636
views
How to handle invalid actions for next state in Q-learning loss
I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values.
...
- 118
1
vote
1
answer
349
views
When should we use CNN instead of MLP?
Is CNN only applicable to time-series data or image data?
When should we use CNN instead of MLP?
- 339
2
votes
1
answer
282
views
What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?
I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.
Here is a screenshot of theorem 3.1.
In the proof of theorem 3.1
...
- 121
5
votes
1
answer
2k
views
Why should one ever use ReLU instead of PReLU?
To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
- 151
4
votes
1
answer
270
views
How to approach a blackjack-like card game with the possibility of cards being counted?
Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.:
You're playing against a dealer with ...
- 41
1
vote
2
answers
166
views
Why does Q-function training not query the Q-function value at unobserved states?
In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that
standard Q-function training does not query the Q-function value at unobserved states, ...
- 225
1
vote
1
answer
352
views
Can I treat "experience" in reinforcement learning as "training data" in statistical learning?
Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning.
Consider the following ...
- 3,990
1
vote
1
answer
439
views
Can I always interpret features as random variables in machine learning safely?
Consider the following statements from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)
Machine learning tasks are usually described in terms of how ...
- 3,990
3
votes
1
answer
450
views
Is it really hard to learn in a stochastic environment?
I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
- 282
8
votes
3
answers
6k
views
What is the difference between the US and global edition of the AIMA book by Russell and Norvig?
The book Artificial Intelligence: A Modern Approach by Russell and Norvig has two editions: global and the US. It looks like these two are generally the same, but have some differences in the order of ...
- 227
-1
votes
1
answer
42
views
What is the borderline between unsupervised learning and regular algorithms?
Unsupervised learning using neural networks is clearly machine learning since it is utilising neural nets.
However, some algorithms, k-means clustering, for example, are considered unsupervised ...
- 1,318
1
vote
1
answer
767
views
When to activate batch normalization and dropout in deep Q-learning?
In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:
When exploring.
When training:
a. When calculating the optimal value of the state reached by an ...
- 111
3
votes
1
answer
246
views
A comparison of Expert Systems and Machine Learning approaches in terms of run-time-efficiency and time/space complexity
For part of a paper I am writing on Clinical Decision Support Systems (computer-aided medical decision making, e.g. diagnosis, treatment), I am trying to compare Expert Systems with systems based on ...
- 25
0
votes
0
answers
29
views
Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?
In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written
The one-step method strengthens only the last action of the sequence of actions that led to the high ...
- 379
0
votes
1
answer
230
views
What is the difference between a vision transformer and image-based relational learning?
I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
- 670
0
votes
1
answer
159
views
Why would the Dice coefficient be more suitable than mutual information when you don't want 0-0 matches to be significant?
I'm confused about the interpretation and assumptions of the Dice coefficient versus the more popular measure mutual information. I'm specifically referencing its use in hierarchical semantic network ...
- 1
1
vote
1
answer
519
views
In deep reinforcement learning, what is this model with state as input and value as output?
I was looking at this implementation for creating an agent for playing Tetris using DeepRL.
This model uses "a state based on the statistics of the board after a potential action. All predictions ...
- 165
1
vote
1
answer
1k
views
Is there any difference between an objective function and a value function?
I found the usage of both objective function and value function in the same context.
Context #1: In the paper titled Generative Adversarial Nets by Ian J. Goodfellow et al.
We simultaneously train G ...
- 3,990
0
votes
0
answers
564
views
Is there any difference between conditional batch normalization and batch normalization except the usage of MLPs for predicting $\beta$ and $\gamma$?
Batch normalization in neural networks uses $\beta$ and $\gamma$ for scaling. The analytical formula is given by
$$\dfrac{x - \mathbb{E}[x]}{\sqrt{Var(X)}}* \gamma + \beta$$
Conditional batch ...
- 3,990
0
votes
1
answer
311
views
Is "kernel" different from "filter" in convolutional neural networks?
Recently I asked a question on how a convolution 2d layer changes an RGB image into a grayscale image. Assume that our task is to convert an RGB image into a grayscale image. I use to believe that ...
- 3,990
0
votes
1
answer
238
views
Does average loss function in GAN training is just an approximation of value function and does not ensure convergence of generator and discriminator?
The value function on which convergence has been proved by the original paper of GAN is
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] + \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$
and ...
- 3,990
0
votes
1
answer
572
views
Is my understanding on "smooth approximation" correct?
Consider the following details regarding Softplus activation function
$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$
SoftPlus is a smooth approximation to the ReLU function and can be
...
- 3,990
2
votes
2
answers
77
views
Why not make the training set and validation set one if their roles are similar?
If the validation set is used to tune the hyperparameters and the training set adjusts the weights, why don't they be one thing as they have a similar role, as in improving the model?
- 43
2
votes
1
answer
860
views
Is there any difference between "image generation" and "image synthesis"?
Generative Adversarial networks (aka GANs) are used for image generation. The phrase image synthesis is also used in literature.
I know that the phrase image generation stands for
An act of ...
- 3,990
12
votes
1
answer
6k
views
In Computer Vision, what is the difference between a transformer and attention?
Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
- 123
0
votes
1
answer
136
views
What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?
What is the advantage of RL compared with the following simple classic algorithm for the MountainCarEnv? Considering that it takes a long time to train the agent ...
- 183
0
votes
0
answers
77
views
Defining states and possible actions in Q learning
I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
- 101
2
votes
0
answers
94
views
Do the terms multi-task and multi-output refer to the same thing in the context of deep learning?
Do the terms multi-task and multi-output refer to the same thing in the context of deep learning (with neural networks)? For example, do neural networks for multi-task learning use multiple outputs?
...
- 339
0
votes
1
answer
2k
views
Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?
I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
0
answers
79
views
Is the main difference between the logistic regression and the perceptron the activation function they use?
I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point.
I'd like to consider the question in terms of the ...
- 217
1
vote
1
answer
131
views
Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
- 573
2
votes
1
answer
4k
views
What are (all) the differences between a neuron and a perceptron?
I know two differences between a neuron and a perceptron
Neuron employs non-linear activation function and perceptron employs only a threshold activation function.
The output of a neuron is not ...
- 3,990