All Questions
664 questions
1
vote
1
answer
24
views
How do neural scaling laws explain the improvements from LSTMs to Transformer based models
I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; ...
- 279
2
votes
1
answer
89
views
Policy performance when the stationary state distribution is not unique in RL
Consider the chainworld above with two actions, move (in red) and stay (in blue). Moving in A is stochastic: the agent moves to B with probability $p$ and to C with probability $1-p$. Moving or ...
- 253
0
votes
0
answers
37
views
When should you use a transformer and when LSTM, GRU and other Neural Networks?
There is a lot of information on the Internet that the transformer is better than RNN in everything, but is it true?
Examples:
«What if I need to translate words?»
«Generate text, images?»
«Play a ...
1
vote
2
answers
40
views
Can the limiting distribution depend on the initial distribution?
I am a bit confused about the definition of limiting distribution in Markov chains.
My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial ...
- 253
2
votes
1
answer
134
views
Ergodic MDP: does it have to be aperiodic?
Puterman defines an ergodic MDP as
if the transition matrix corresponding to every deterministic stationary policy consists of a single recurrent class.
If the transition matrix is recurrent, it ...
- 253
1
vote
1
answer
50
views
Conditions for the existence of stationary state distribution
If the Markov chain induced by the policy is ergodic, then the state distribution under the policy is stationary.
Are there some conditions/properties (sufficient or necessary, on the MDP or on the ...
- 253
1
vote
2
answers
93
views
Why are ergodic MDPs also communicating?
An MDP is ergodic if the Markov chain induced by any policy is
ergodic, which means any state is reachable from any other state by
following a suitable policy. [Source]
The part after "which ...
- 253
1
vote
3
answers
64
views
How do 2-player games fit under the MDP framework of reinforcement learning?
I am confused about the theoretical framework of reinforcement learning. For supervised learning, there seems to be a clear theoretical framework, e.g. as described by Wikipedia here. I am unclear ...
- 111
0
votes
2
answers
55
views
Is there any actual difference between these 2 definitions of a state value function?
The definition of the value function in TRPO paper is
\begin{align}
V_\pi(s_t) &= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt]
a_t &\sim \pi(...
- 139
1
vote
0
answers
28
views
Linear / Quadratic Program for Solving MDPs
Aside from value iteration, we can use the following linear program to solve the optimal value function of an MDP.
I am planning to put some constraints on ...
- 41
3
votes
2
answers
451
views
Can reinforcement learning rewards be a combination of current and new state?
I'm structuring the reward function for my RL agent and considering a combination of both the current state and the new state after taking an action. From what I understand, this is possible based on ...
- 139
1
vote
1
answer
36
views
Do State Variables in RL Models Need Direct Update Equations?
I'm working on a simulation model using RL to optimize an objective function. I'm trying to understand if I need to select my state variables such that I can write state update equations for each one ...
- 139
0
votes
1
answer
108
views
Does Machine Learning focus on discriminative AI while Deep Learning also focus on generative AI?
I know that Deep Learning is subset of Machine learning
But is it correct that classical ML algorithms mainly focus on implementing Discriminative AI while DL algorithms implement both Generative AI ...
- 181
0
votes
1
answer
43
views
Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?
Let's hypothesize a scenario where some of the records (si, ai, ri, si+1) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random ...
- 1
0
votes
1
answer
50
views
Are there problems where the optimal policy is stochastic?
I know that in a MDP there always exists a unique optimal deterministic policy. Does a statement like this also exist for optimal stochastic policies? Is there also always a unique optimal stochastic ...
- 139
0
votes
2
answers
107
views
What do we mean by "AI is correlated"?
From Wikipedia
Causal AI is a technique in artificial intelligence that builds a causal model and can thereby make inferences using causality rather than just correlation. One practical use for ...
- 117
-1
votes
1
answer
91
views
Violation of Markov property
Consider the following cases:
a-) In solving an episodic problem we observe that all trajectories from the start state to the goal state pass through a particular state exactly twice.
b-) In solving ...
- 1,115
0
votes
0
answers
35
views
Q-Learning conditions for convergence and ergodicity
Q-learning is guaranteed to convergence if the learning rate satisfies the Robbins-Monro conditions and if every state-action pair is visited infinitely often.
Regarding the latter, does it mean that ...
- 253
0
votes
1
answer
136
views
Value iteration in a Grid World Example
I am facing some confusion regarding the calculation of the values for the states in a grid world.
Given that this is my grid world, where the there is a reward for going to F of +1, and no other ...
0
votes
1
answer
43
views
Optimality of two policies versus variance of returns from a state
If for an MDP there exist two optimal policies, it may be possible that the variance of their returns are different for a given state.
This is correct, right?
- 1,115
0
votes
0
answers
21
views
Properties of an example environment
Let us consider the following problem.
A student is developing a robot that can roll a die. The robot grips and releases the die with an arm that can also twist, and uses two cameras that can see the ...
- 1,115
0
votes
0
answers
45
views
In Markov Decision Process, how to understand the calculation of the average length of episode?
In the Sec. 13.2 of RL: An Introduction (Sutton & Barto), the concept of average episode length is discussed for both episodic MDP and continuing MDP.
In an episodic MDP, the average length of an ...
2
votes
1
answer
47
views
MDP Average Reward independent of Initial State
Asking this question of mine in MathOverflow here since AI StackExchange appears to be a more appropriate place.
Consider a Markov Decision Process where the state space $S$ and the action space $A$ ...
- 121
2
votes
2
answers
556
views
Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?
I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation ...
- 23
1
vote
1
answer
21
views
Why are there up to $m^2$ action values when we consider the complexity of DP based on $q(x,u)$?
Please see slide 32 in the following lecture slides on DP:
https://groups.uni-paderborn.de/lea/share/lehre/reinforcementlearning/lecture_slides/built/Lecture03.pdf
Let $m$ the size size of action ...
- 1,115
1
vote
2
answers
52
views
Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem?
Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem? I need a clear explanation.
- 1,115
0
votes
1
answer
24
views
MDP and a given policy and the correctness of the state-value function
Is the following statement correct?
"For an MDP and a given policy, the Bellman equation can be used to check the correctness of the state-value function."
- 1,115
0
votes
0
answers
21
views
Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?
https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
- 3,267
1
vote
2
answers
1k
views
What is the difference between densenet and resnet?
Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating.
The Densenet paper appears to be ...
- 233
0
votes
1
answer
39
views
Is the sequence 1-1-2-3-Exit possible in the following Markov reward process?
Is the sequence 1-1-2-3-Exit possible in the following Markov reward process?
The probability of transitioning from state 1 to itself is 0.
Source: https://maelfabien.github.io/rl/RL_2/#markov-process-...
- 1,115
2
votes
2
answers
94
views
If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite? [duplicate]
In chapter 2 of Barto and Sutton's RL book, the four argument probability function $p: S \times R \times S \times A \to [0,1]$ is reduced to three arguments $p: S \times S \times A \to [0,1]$ as ...
0
votes
2
answers
56
views
What is the logic in including/not including subscript $\pi$ in in "E" for value functions? [closed]
Here are two relations for value functions:
Eq.1: $v_{\pi}(s)=E_{\pi}[q_{\pi}(S_t, A_t)|S_t=s]$
Eq.1: $q_{\pi}(s,a)=E[R_{t+1}+\gamma v_{\pi}(S_{t+1})|S_t=s, A_t=a]$
Question: Why is there subscript $\...
- 1,115
0
votes
0
answers
27
views
What does the term "expected leaf node" in this exercise from Sutton-Barto mean?
What does the term "expected leaf node" in the Exercise below from Sutton-Barto mean?
- 1,115
-1
votes
1
answer
197
views
Is there any example of a Markov Decision Process (MDP) with infinite number of states?
I was learning fundamentals of reinforcement learning from various sources like coursera, Udacity, ...
- 157
1
vote
1
answer
106
views
When is it non-Markovian?
Several months ago, I was writing for class. I claimed an environment was non-Markovian because it would take several states to de-alias some positions in the grid world. I was corrected that it was ...
- 1,559
1
vote
1
answer
72
views
How to properly model the MDP of a weighted graph with the constraint of only visiting each vertex once (and not get stuck in infinite loops)?
I'm trying to model a MDP to traverse a complete weighted graph (i.e. all vertex are connected). The states, and also the actions (i.e. S=A), are the vertex of the weighted graph. The transition ...
1
vote
1
answer
98
views
Is there any advantage of genetic algorithm (or programming) over Neural Networks? [closed]
I am planning to switch from neural networks to genetic algorithms (GA) and programming (GP).
One of the main hassles of working with neural networks is that it requires a large amount of training ...
- 339
1
vote
0
answers
76
views
Solving MDP as linear program: why minimize the sum of the states' values?
This is a follow-up question to the answer to How can we use linear programming to solve an MDP?
Quick recap: the $max$ operators that appear in the Bellman optimality equations can be turned into a ...
- 111
0
votes
1
answer
331
views
Can Q-learning rewards and next states be non-deterministic?
I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows:
...
0
votes
2
answers
61
views
Should I define my problem as image segmentation or detection?
I have a problem and have to decide wether it's an object detection or object segmentation problem. I want to use Yolov8 for training. We already have hundrets of images but they aren't labeled yet. ...
- 113
1
vote
1
answer
1k
views
When to use Pruning, Quantization , Distillation and others when optimizing speed
I want to understand how to optimize models for inference speed and am seeking some advice and best practices for the same.
I am a little bit aware of the concepts of pruning, quantization, and ...
- 789
4
votes
2
answers
3k
views
What are the differences between seq2seq and encoder-decoder architectures?
I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user78615
1
vote
1
answer
133
views
Why are these two implementations of the $\epsilon$-greedy policy different?
According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:
$$
\pi(a|s) =
\begin{cases}
\frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
- 195
2
votes
1
answer
541
views
What are the similarities between Q-learning and Value Iteration?
This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
- 23
1
vote
0
answers
70
views
When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?
When reading the following paper(page 4): An Approximate Dynamic Programming Approach
for Dual Stochastic Model Predictive Control
I could see that they were able to unnest the minimization's in the ...
- 11
1
vote
0
answers
34
views
How can I prove that early termination in an MDP state is valid?
I have an MDP $M$ with transition function $p$, states $S$ and a state $s^0 \in S$. $s^0$ has the property that both the most optimistic trajectory (highest expected reward) and the most pessimistic ...
- 111
1
vote
0
answers
71
views
Is this a bandit problem or a MDP?
I am trying to understand if this problem can be casted both as a bandit problem as well as an MDP.
Lets assume that we are trying to optimize sales $y_t$ based on investments $x_{1, t}, x_{2, t}$ ...
- 53
1
vote
1
answer
48
views
Is there validation data in K-fold cross-validation?
We know that in machine learning the dataset is divided into 3 parts: training data, validation data and test data.
On the other hand, K-fold cross-validation is defined as follows:
the dataset is ...
- 1,115
2
votes
1
answer
132
views
UCB, Thompson sampling etc seems myopic/greedy for bandits?
When considering multi-armed bandits in different formats, UCB, $\epsilon$-greedy, thompson sampling etc seems so greedy/myopic in the sense that it solely considers reward for the current timestep.
...
- 53
0
votes
1
answer
65
views
Why is R(s) more restrictive than R(s, a) in an MDP?
I am quite new to RL. I would like to know why a state-dependent reward function R(s) is more restrictive than a state-action-dependent reward function R(s, a)? And why is it that a policy can be ...
- 11