31
votes
Accepted
Where can I find the proof of the universal approximation theorem?
There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different ...
- 41.4k
18
votes
Are there other approaches to deal with variable action spaces?
Does anyone know any paper regarding this subject?
I'm not familiar with any off the top of my head. I do know that the vast majority of Reinforcement Learning literature focuses on settings with a ...
- 10.5k
15
votes
Accepted
Can supervised learning be recast as reinforcement learning problem?
Any supervised learning (SL) problem can be cast as an equivalent reinforcement learning (RL) one.
Suppose you have the training dataset $\mathcal{D} = \{ (x_i, y_i \}_{i=1}^N$, where $x_i$ is an ...
- 41.4k
14
votes
Why doesn't Q-learning converge when using function approximation?
Here's an intuitive description answer:
Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
- 9,392
12
votes
Is there a way to understand neural networks without using the concept of brain?
tl;dr I always like to think of Neural Networks as a generalization of logistic regression.
I too don't like that, traditionally, when introducing Neural Networks, books start with biological ...
- 3,213
9
votes
Why doesn't Q-learning converge when using function approximation?
As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
- 10.5k
9
votes
Can a neural network with linear activation functions produce a connection of linear functions?
A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.
If we had multiple linearly activated layers, each feeding into ...
- 3,857
8
votes
Accepted
Why can't neural networks learn functions outside of the specified domains?
The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way ...
- 2,389
6
votes
Where can I find the proof of the universal approximation theorem?
"Modern" Guarantees for Feed-Forward Neural Networks
My answer will complement nbro's above, which gave a very nice overview of universal approximation theorems for different types of ...
- 564
6
votes
Which functions can't neural networks learn efficiently?
One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible.
"A feedforward network with a single layer is ...
- 353
6
votes
Accepted
Smallest possible network to approximate the $sin$ function
Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
- 420
6
votes
Accepted
What is the relation between the context in contextual bandits and the state in reinforcement learning?
The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in ...
- 41.4k
6
votes
Accepted
Why policy improvement theorem can't be applied in case of function approximation?
As referenced in Sutton's RL book regarding policy improvement theorem (PIT):
The policy improvement theorem applies to the two policies that we considered at the
beginning of this section: an ...
- 5,447
5
votes
Accepted
What makes multi-layer neural networks able to perform nonlinear operations?
Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.
- 578
5
votes
Is a multilayer perceptron a recursive function?
Inherently, no. The MLP is just a data structure. It represents a function, but a standard MLP is just representing an input-output mapping, and there's no recursive structure to it.
On the other ...
- 9,392
5
votes
Accepted
Why is there a sigmoid function in the hidden layer of a neural network?
Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
layer_output = Weights.layer_input + bias
...
5
votes
Accepted
Why are only neural networks (and not SVMs, for example) used for reinforcement learning?
The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally ...
- 33.3k
4
votes
Is there a way of converting a neural network to another one that represents the same function?
To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. ...
- 416
4
votes
Are there other approaches to deal with variable action spaces?
Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action ...
- 563
4
votes
Accepted
Which machine learning models are universal function approximators?
Support vector machines
In the paper A Note on the Universal Approximation Capability of Support Vector Machines (2002) B. Hammer and K. Gersmann investigate the universal function approximation ...
- 41.4k
4
votes
Accepted
Why do all states appear identical under the function approximation in the Short Corridor task?
You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
3
votes
Why don't people use projected Bellman error with deep neural networks?
I have found some clues in Maei's thesis (2011): “Gradient Temporal-Difference Learning Algorithms.”
According to the thesis:
GTD2 is a method that minimizes the projected Bellman error (MSPBE).
...
- 520
3
votes
Why doesn't Q-learning converge when using function approximation?
There are three problems
Limited capacity Neural Network (explained by John)
Non-stationary Target
Non-stationary distribution
Non-stationary Target
In tabular Q-learning, when we update a Q-value, ...
- 131
3
votes
Accepted
How can my Q-learning agent trained to solve a specific maze generalize to other mazes?
I'm going to assume here that you're using the standard, basic, simple variant of $Q$-learning that can be described as tabular $Q$-learning, where all of your state-action pairs for which you're ...
- 10.5k
3
votes
Is a multilayer perceptron a recursive function?
Sure, you can define plenty of things we don't generally need to regard as recursive as so. An MLP is just a series of functions applied to its input. This can be loosely formulated as
$$ o_n = f(o_{...
- 326
3
votes
Is there a way to understand neural networks without using the concept of brain?
One way to view a neural network is as a series of linear transformations.
You take a bunch of data points and look at it from a different perspective from a different space. You apply some non ...
3
votes
Accepted
Is there a possibility that there is no relationship between some inputs and outputs?
Of course, it's possible to define a problem where there is no relationship between input $x$ and output $y$. In general, if the mutual information between $x$ and $y$ is zero (i.e. $x$ and $y$ are ...
- 286
3
votes
Accepted
Why are neural networks preferred to other classification functions optimized by gradient decent
You can indeed fit a polynomial to your labelled data, which is known as polynomial regression (which can e.g. be done with the function numpy.polyfit). One ...
- 41.4k
3
votes
Accepted
How do we derive the expression for average reward setting in continuing tasks?
We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited ...
- 5,030
3
votes
Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?
First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many ...
- 5,030
Only top scored, non community-wiki answers of a minimum length are eligible