Neil Slater
  • Member for 5 years, 4 months
  • Last seen this week
  • Durham, United Kingdom
Does a second-order fully-connected layer have any uses?
Accepted answer
1 votes

Is this a valid implementation of second-order regression? No, but it is not far off. To perform a full second-order regression, you will need all terms for $x_{i,j}x_{i,k}$ where the first index is ...

View answer
What is the stride information of an image referring here?
1 votes

Do they mean the strides that are related to the CNN, pooling, etc., or are they referring to any other stride information? The stride referred to by the quote "only plays with the size and ...

View answer
Why do terms in the computation of state space size scale exponentially?
Accepted answer
1 votes

Intuitively, I feel like if there are 30 foods, each with 2 states, then that is 60 states, no $2^{30}$. Let's try it with 3 pellets. If you are right there would be $2 \times 3 = 6$ states, if the ...

View answer
Does reinforcement learning lend itself well to changes in the environment due to external factors?
1 votes

In the question, you are not describing the environment changing. Instead, there is a fixed 20% chance of a bad weather event each year. Such events can me modelled as a static environment with ...

View answer
Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?
Accepted answer
4 votes

It is important to note that the graph shows reward received during training. This includes rewards due to exploratory moves, which sometimes involve the agent falling off the cliff, even if it has ...

View answer
Why is old/off-policy data harmful to on-policy/online RL?
1 votes

there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent ...

View answer
Why is the performance of my neural network to predict if the mean of a randomly generated tensor is greater than $0.5$ not good?
0 votes

As you made this experiment available on Colab, I was able to test my thoughts on it, which was handy. First, the simple "fix" is to run many epochs. Eventually even your relatively small ...

View answer
Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?
Accepted answer
0 votes

As I understand it, the Bellman equation assumes the setting to be deterministic, meaning that, if you're in state $s_t$ as you take action $a_1$, you should always reach the same $s_{t+1}$. This is ...

View answer
Why are we choosing more than 1 action in SARSA?
1 votes

Why are we choosing more than 1 action in SARSA? There is never a state where more than one action is chosen. The appearance of two Choose statements is an artifact of the loop design and variable ...

View answer
Are we choosing the same action in every step in SARSA?
1 votes

Do we only select one action at the very beginning and then we always choose the same action for each step? No. The pseudocode is clear on this, by using the word Choose and referencing a policy. If ...

View answer
Are the two policies in SARSA for choosing an action the same?
1 votes

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode. ...

View answer
Can neural networks learn noise?
1 votes

What explains the apparent 'mirroring' of the graphs on the RHS, The model starts untrained and no better than random guessing (the baseline). As the training progresses, the model does better than ...

View answer
Is there a mathematical formalism to deal with a missing reward signal?
2 votes

Your setting (of randomly dropping out reward signals) impacts expected future reward by multiply everything by a common factor $(1-\epsilon)$. As reinforcement learning (RL) control is based on ...

View answer
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
Accepted answer
1 votes

The tanh functions within the cell represent cell output or cell state. These are the values that either get passed on to other layers, or within the layer to the next time step. In theory, other ...

View answer
Is it a bad practice to use cumulative rewards in reinforcement learning
Accepted answer
1 votes

Reinforcement learning already has the objective of maximising the sum of future expected reward. By making each reward the sum of all previous rewards, you will make the the difference between good ...

View answer
When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
Accepted answer
5 votes

The core differences between using $V(s)$ or $Q(s,a)$ are: $V(s)$ cannot be used stand-alone to decide a policy. You either need a separate policy function $\pi(a|s)$ that it is the value function for,...

View answer
How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
Accepted answer
1 votes

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough. A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for ...

View answer
Can I flip a video to generate more data for action recognition?
0 votes

Probably flipping a video left/right will be OK and useful for your case. When considering data augmentation approaches, you should think about two things that may prevent it working: Could the ...

View answer
How does this TD(0) off-policy value update formula work?
Accepted answer
2 votes

This would mean we decrease the value of this state. Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy ...

View answer
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
Accepted answer
1 votes

Can't it be that the optimal policy thinks a state isn't that good and gives him a low value but perform best in comparison with other policies which have higher values for this state? No, this is ...

View answer
When can I call an entity a hyperparameter?
2 votes

Is it okay to call anything that needs to be learned outside the training algorithm a hyperparameter? I think so, yes. Personally, I would reserve the term to discuss values that I could choose ...

View answer
How should I define the reward function for a stock trading-like game?
1 votes

how should the reward scheme be for a game like this? i.e., whether one action is good or not depends on other actions taken before it? The reward scheme should always be a "natural" ...

View answer
Reinforcement Learning method suitable for a large discrete action space with high sample efficiency
0 votes

My understanding of your environment is: The batch number $b$ is the same thing as a time step $t$. Each batch is associated with a single static representation of the environment, the agent makes ...

View answer
Can RL still learn in a scenario where current state and the next state are independant?
1 votes

You don't have a full reinforcement learning problem, but appear to have a context-free k-armed bandit problem: The start state at time $t$ is essentially irrelevant to the problem. It does not ...

View answer
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Accepted answer
6 votes

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model. What you can usually drop when you have large amounts of ...

View answer
What sort of out-of-the-box technology could be used to create work similar to artist Refik Anadol?
0 votes

For a Google term you could use "computational creativity". It covers a wide range of ideas, and the artist here is not using one single tool or approach. There are clearly a range of ...

View answer
Is there any recommended way to perform pooling in this context?
Accepted answer
2 votes

This one is a bit crazy: pool1 = nn.AvgPool3d(kernel_size = (361, 1, 1), stride= 1) because it averages large numbers of the features at once. Very little information about individual features will ...

View answer
What do people refer to when they use the word 'dimensionality' in the context of convolutional layer?
1 votes

The dimensionality used to discuss convolutional layers in CNNs is based on the dimensionality of the input without considering channels. 1D CNNs might process raw audio sources (mono or stereo), ...

View answer
How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?
Accepted answer
0 votes

The accuracy level is already very high, but how to motivate the agent to find the target as quickly as possible? You already are, in two different ways: A penalty (negative reward) for each time ...

View answer
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
1 votes

The reward is given only at the end of the episode (or when there is timeout there is no reward) This is a common case. E.g. winning a board game, or reaching a goal state. How could we learn the ...

View answer
1
2 3 4 5
20