26 votes
Accepted

Where can I find the proof of the universal approximation theorem?

There are multiple papers on the topic because there have been multiple attempts to prove that neural networks are universal (i.e. they can approximate any continuous function) from slightly different ...
  • 36.3k
18 votes

Are there other approaches to deal with variable action spaces?

Does anyone know any paper regarding this subject? I'm not familiar with any off the top of my head. I do know that the vast majority of Reinforcement Learning literature focuses on settings with a ...
  • 9,659
13 votes

Why doesn't Q-learning converge when using function approximation?

Here's an intuitive description answer: Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
13 votes
Accepted

Can supervised learning be recast as reinforcement learning problem?

Any supervised learning (SL) problem can be cast as an equivalent reinforcement learning (RL) one. Suppose you have the training dataset $\mathcal{D} = \{ (x_i, y_i \}_{i=1}^N$, where $x_i$ is an ...
  • 36.3k
12 votes

Is there a way to understand neural networks without using the concept of brain?

tl;dr I always like to think of Neural Networks as a generalization of logistic regression. I too don't like that, traditionally, when introducing Neural Networks, books start with biological ...
  • 3,133
8 votes

Can a neural network with linear activation functions produce a connection of linear functions?

A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network. If we had multiple linearly activated layers, each feeding into ...
8 votes
Accepted

Why can't neural networks learn functions outside of the specified domains?

The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way ...
  • 2,329
7 votes

Why doesn't Q-learning converge when using function approximation?

As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
  • 9,659
6 votes

Which functions can't neural networks learn efficiently?

One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible. "A feedforward network with a single layer is ...
6 votes
Accepted

Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
  • 420
5 votes

Where can I find the proof of the universal approximation theorem?

"Modern" Guarantees for Feed-Forward Neural Networks My answer will complement nbro's above, which gave a very nice overview of universal approximation theorems for different types of ...
  • 525
5 votes
Accepted

What makes multi-layer neural networks able to perform nonlinear operations?

Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.
5 votes

Is a multilayer perceptron a recursive function?

Inherently, no. The MLP is just a data structure. It represents a function, but a standard MLP is just representing an input-output mapping, and there's no recursive structure to it. On the other ...
5 votes
Accepted

Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias ...
4 votes

Is there a way of converting a neural network to another one that represents the same function?

To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. ...
4 votes

Are there other approaches to deal with variable action spaces?

Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action ...
4 votes
Accepted

Which machine learning models are universal function approximators?

Support vector machines In the paper A Note on the Universal Approximation Capability of Support Vector Machines (2002) B. Hammer and K. Gersmann investigate the universal function approximation ...
  • 36.3k
4 votes
Accepted

Why do all states appear identical under the function approximation in the Short Corridor task?

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
4 votes
Accepted

What is the relation between the context in contextual bandits and the state in reinforcement learning?

The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in ...
  • 36.3k
3 votes

Why don't people use projected Bellman error with deep neural networks?

I have found some clues in Maei's thesis (2011): “Gradient Temporal-Difference Learning Algorithms.” According to the thesis: GTD2 is a method that minimizes the projected Bellman error (MSPBE). ...
  • 510
3 votes

Why doesn't Q-learning converge when using function approximation?

There are three problems Limited capacity Neural Network (explained by John) Non-stationary Target Non-stationary distribution Non-stationary Target In tabular Q-learning, when we update a Q-value, ...
3 votes

Is a multilayer perceptron a recursive function?

Sure, you can define plenty of things we don't generally need to regard as recursive as so. An MLP is just a series of functions applied to its input. This can be loosely formulated as $$ o_n = f(o_{...
  • 326
3 votes
Accepted

How can my Q-learning agent trained to solve a specific maze generalize to other mazes?

I'm going to assume here that you're using the standard, basic, simple variant of $Q$-learning that can be described as tabular $Q$-learning, where all of your state-action pairs for which you're ...
  • 9,659
3 votes

Is there a way to understand neural networks without using the concept of brain?

One way to view a neural network is as a series of linear transformations. You take a bunch of data points and look at it from a different perspective from a different space. You apply some non ...
3 votes
Accepted

Is there a possibility that there is no relationship between some inputs and outputs?

Of course, it's possible to define a problem where there is no relationship between input $x$ and output $y$. In general, if the mutual information between $x$ and $y$ is zero (i.e. $x$ and $y$ are ...
3 votes
Accepted

Why are neural networks preferred to other classification functions optimized by gradient decent

You can indeed fit a polynomial to your labelled data, which is known as polynomial regression (which can e.g. be done with the function numpy.polyfit). One ...
  • 36.3k
3 votes
Accepted

How do we derive the expression for average reward setting in continuing tasks?

We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited ...
3 votes

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many ...
3 votes

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in CB, compared to states in RL? In terms of its place in the description of Contextual Bandits and Reinforcement Learning, context in CB is ...
  • 25.4k
3 votes
Accepted

Why are only neural networks (and not SVMs, for example) used for reinforcement learning?

The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally ...
  • 25.4k

Only top scored, non community-wiki answers of a minimum length are eligible