# Tag Info

18

An algorithm is sample efficient if it can get the most out of every sample. Imagine learning trying to learn how to play PONG for the first time. As a human, it would take you within seconds to learn how to play the game based on very few samples. This makes you very "sample efficient". Modern RL algorithms would have to see $100$ thousand times more data ...

14

The paper's authors needed to implement their models anyway in order to conduct their experimentations, so why not publish the implementation? Some papers and authors actually provide a link to their own implementation, but most of the papers (that I have read) don't provide it, although some third-party implementations may already be available on Github (...

7

The two tech reports below both call RNNs explicitly "recurrent net(work)s". One of them predates the paper mentioned in the accepted answer. Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, ...

6

When crossover happens and one parent is fitter than the other, the nodes from the more fit parent are carried over to the child. This is the case as disjoint and excess genes are only carried over from the fittest parent. Here's an example: // Node Crossover Parent 1 Nodes: {[0][1][2]} // more fit parent Parent 2 Nodes: {[0][1][2][3]} Child Nodes: {[0]...

5

It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades. Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and ...

5

Sample Efficiency denotes the amount of experience that an agent/algorithm needs to generate in an environment (e.g. the number of actions it takes and number of resulting states + rewards it observes) during training in order to reach a certain level of performance. Intuitively, you could say an algorithm is sample efficient if it can make good use of every ...

5

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment. In general I would say that there are two issues related to sample efficiency: 1 the size of the 'action'+'environment states' space 2 the ...

5

Someone can argue to some human adequate reasons, but there is a bad trend of falsified results in deep learning research papers that propose some nowel solutions or even update state-of-the-art model performance. And that's not just a few papers that lie, it's a large portion of them. And the reason for that is even more sad - most of so-called deep ...

5

The first reason described in nbro's answer can definitely be an important one; authors may have implemented their software using code that they can't share. There's a lot of research coming out of companies (large and small), and they may use all sorts of proprietary libraries that were built in the company and cannot be distributed outside. As described in ...

4

Examining the architecture of the DNC indeed shows many similarities to the LSTM. Consider the diagram in the DeepMind article that you linked to: Compare this to the LSTM architecture (credit to ananth on SlideShare): There are some close analogs here: Much like the LSTM, the DNC will perform some conversion from input to fixed-size state vectors (h and ...

4

Go predictions were included in the paper: The experts are far from infallible. They predicted that AI would be better than humans at Go by about 2027. (This was in 2015, remember.) SOURCE: Experts Predict When Artificial Intelligence Will Exceed Human Performance (MIT Tech Review)

4

A non-starving policy is a (behavior) policy that is theoretically guaranteed to visit each state and take all possible actions from each state an infinite number of times, so that to always update $Q(s, a)$, $\forall s, \forall a$, an infinite number of times. In the context of off-policy prediction, this criterion implies that any trajectory will have no ...

4

In language theory, there are generally several admitted levels that can be studied in relation with one another or independently. The semantic level is the one dealing with the meaning of the text ("semantic" comes from the greek and means "to signify"). The semantic level is therefore generally independent from the syntax and even the language used to ...

4

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get gold looking object. When you remove these priors you can see a human is worse at learning these games. (link) Some experiments they tried in the study to ...

3

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me. I'll disregard expectation here to make things simpler. First, note that policy $\pi$ depends on parameter vector $\phi$ and function $f_\phi(\epsilon_t;s_t)$, and value function $Q$ depends on parameter ...

3

If the gradients are noisy (does this mean that in some dimension we have small and in some high curvature or that error noise differs for very similar values of w?) Gradients being noisy means that they are "inconsistent" across different epochs / training steps. With that I mean that they'll sometimes point in one direction, later in a different ...

3

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward shaping (PBRS). The issue is actually worse than you describe---it's more general than $s = s'$. In particular, the issue presents itself differently based on the ...

3

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the state $s$, that you expect to be able to get high future returns from that state. If you then continuously take actions that keep you in the same state $s$, it ...

3

How would you implement this "Number of Steps" cost? What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this). As ...

3

I recommend you focus on quality over quantity. Publishing a paper will boost your reputation and make you more recognised within your academic field (AI), however this is only if the paper provides useful insights into an important issue. Your paper is more likely to be accepted if it is well written and easy to understand, stimulates new important ...

3

locality of pixel dependencies probably means that neighboring pixels tend to be correlated, while faraway pixels are usually not correlated. This assumption is usually made in several image processing techniques (e.g. filters). Of course, the size and the shape of the neighborhood could vary, depending on the region of the image (or whatever), but, in ...

3

The square brackets $[]$ in $[\tau_{ij}]^\alpha$ and $[\eta_{ij}]^\beta$ may be just a way of emphasing that the elements $\tau_{ij} \in \mathbb{R}$ and $\eta_{ij} \in \mathbb{R}$ of respectively the matrices $\mathbf{\tau} \in \mathbb{R}^{n \times n}$ and $\mathbf{\eta} \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the graph) are ...

3

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because many of the code implementations on github focus on trajectories starting from the same state. Hope this helps everyone !

3

3

I think the answer depends very much on why you are reading the paper, what are you trying to get out of it? There are plenty of papers that I "read" (or often really just quickly skim through) where I'll definitely not understand all the math. More often than not, this will be because I don't actually care to deeply understand it. There is plenty ...

3

Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will always give you a better-performing model. However, since you can retrain many times to get a distribution of models, you can use a statistical test like the T-Test to say "algorithm A usually produces a better model ...

3

Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves that theoretically the algorithm achieves the desired function. Without convergence, we have no guarantees that the value function will be accurate or the policy ...

3

This answer assumes that you only have a problem with this notation from the article: $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ This is a standard notation, used in many disciplines, for defining a function and its input and output domains. It is a bit like the method signature for the function - it does not fully define it, but does ...

3

Recently arxiv.org added a Code Tab towards the end of paper descriptions. Which contains links to both the official and community code. I don't know if this is the case for all the papers or not till know. But I'm sure it'll be extended to all the papers in a short while.

2

One important consideration here: in the last decade or two the machine learning and artificial intelligence fields, which contains the majority of reinforcement learning work, researchers have considered conferences to be the more impactful publishing venues than journals. The particular venue a researcher chooses depends on the data and/or application ...

Only top voted, non community-wiki answers of a minimum length are eligible