Which is more important, doubt or reinforcement?

Question

Reinforcement?

We hear much about reinforcement, which is, in my opinion a poor choice of a term to describe a type of artificial network that continues to acquire or improve its behavioral information in natura (during operations in the field). Reinforcement in learning theory is a term used to describe repetitious incentivization to increase the durability of learned material. In machine learning, the term has been twisted to denote the application of feedback in operations, a form of re-entrant back propagation.

Corrective Signaling

Qualitatively, corrective signaling in field operations can supply information to a network to make only two types of functional adjustments.

• Adjustments to what is considered the optimum, beginning with the optimum found during training prior to deployment
• Testing of entirely new areas of the parameter space for hint of new optima that have formed, any of which might currently qualify or soon qualify as the global optimum.

(By optima and optimum, we mean minima and global minimum in the surface that describes the disparity between ideal system behavior and current system behavior. This surface is sometimes termed the error surface, applying an over-simplifying analogy from the mathematical discipline of curve fitting.)

The Importance of Doubt

The second of the two above could aptly be termed doubt.

Perhaps all neural nets should have one or more parallel doubting networks that can test remote areas of the search space for more promising optima. In a parallel computing environment, this might be a matter of provisioning and not significantly reduce the throughput of the primary network, yet provide a layer of reliability not found without the doubtful parallel networks.

What Shows More Intelligence?

Which is more important in actual field use of AI? The ability to reinforce what is already learned or the ability to create a minority opinion, doubt the status quo, and determine if it is not a more appropriate behavioral alternative than that which was reinforced.

A Helpful Pool of Water Analogy

During a short period of time, a point on the surface of the water may be the lowest point in a pool. With adjustments based on gradient (what is so inappropriately called reinforcement) the local well can be tracked so the low point can be maintained without any discrete jumps to other minima in the surface. However the local well may cease being the global minimum at some point in time, whereby a new search for a global minimum must ensue.

It may be that the new global minimum is across several features on the surface of the pool and cannot be found with gradient descent.

More interestingly, the appearance of new global minima can be tracked and reasonable projections can be made such that discrete and substantial jumps in parametric state can be accomplished without large jumps in disparity (where the system misbehaves badly for a period).

Circling Back to the Question

Which is more important, doubt or reinforcement?

Answer 1

Which is more important, doubt or reinforcement?

The single-sentence answer to this would be: it depends.

The core of this question seems to be very closely related to the well-known trade-off between exploration (similar to how you describe "doubt") and exploitation (similar to how you describe "reinforcement"). It is almost never the case that someone declares one of those two to be more important than the other, and only tries to pursue the "most important one". There is no single axis that measures "importance", no single line of numbers such that we can place "doubt" on one point, "reinforcement" on the other, and declare that the biggest number is the most important one. We almost always want a balance between exploration and exploitation. They are, almost always, both important.

Now, in some extreme cases, only one of the two may be important. For example:

• If you have an environment where you can generate experience and evaluate policies completely free of any costs, you'll want to prioritize exploration / doubt. This is almost never the case though, you'll realistically always have at least time as a cost.
• If you have a situation where you care very much about your performance right now, there's no point in doubting it too much or exploring too much. Consider, for example, DeepMind's AlphaGo team a few hours before their match against world-class human player Lee Sedol in 2016. At such a point in time, I highly doubt they'll be interested in exploring wildly different sets of parameters from the ones they have found during training so far; they won't have time anymore to thoroughly evaluate them, they'll want to stick to what they have, which they know works fairly well.

Also note that sometimes, you need to stick to something that already works well somewhere if you want to be able to realistically learn something new somewhere else:

• In a large Markov Decision Process (e.g. Montezuma's Revenge), once you've already learned a good policy near the initial state, you don't want to explore too much anymore around that initial state because you won't reach interesting new states. You need to exploit for a while first such that you actually reach new states where it becomes interesting to explore again.
• A similar situation, but now more closely related, using your terminology of "doubt" and viewing the space of all possible sets of values for all parameters of a large Neural Network as the space that we're searching in; suppose that we're learning a policy for an Atari game, with pixels as input. We first have a few Convolutional layers, then a few Relu's, etc., the standard setup. Intuitively, we expect the first few layers to "learn" how to "understand" the images, and transform the raw pixels to encodings that can be used in an efficient manner for the last few layers to compute a good policy / value estimates. If the first few layers already do a good job at transforming the raw pixel inputs to something more useful, we generally don't want to "doubt" those layers too much anymore, we don't suddenly want to jump to a completely different set of parameters anymore. If we suddenly completely change the parameters in those layers, we'll know for sure that the later layers will also be completely messed up. We'll simply have created a significantly more difficult learning problem for ourselves when it comes to optimizing the last few layers.

---

Some other notes:

---

There are lots of references to "networks" in the question, for example the quote below:

We hear much about reinforcement, which is, in my opinion a poor choice of a term to describe a type of artificial network that continues to acquire or improve its behavioral information in natura (during operations in the field).

Note that Reinforcement (learning) does not necessarily have to involve any kinds of artificial (neural or otherwise) networks at all. There's also tabular RL, and RL with function approximation using non-network function approximators (e.g. linear).

---

Under the "What shows more intelligence?" header, you write:

The ability to reinforce what is already learned

That is not an accurate description of what was previously described as

Adjustments to what is considered the optimum, beginning with the optimum found during training prior to deployment

which, in turn, I suppose is intended to describe the kinds of updates that are typically performed using variants of the Bellman optimality equation (value-based methods) or variants of REINFORCE (policy gradient methods). These methods do not just "reinforce what is already learned". They can lead to learning completely new policies from completely new experiences.

---

This idea:

Perhaps all neural nets should have one or more parallel doubting networks that can test remote areas of the search space for more promising optima. In a parallel computing environment, this might be a matter of provisioning and not significantly reduce the throughput of the primary network, yet provide a layer of reliability not found without the doubtful parallel networks.

sounds a lot to me like using Evolutionary Search to optimize the parameter of a network. Some interesting blog posts from Uber (with references to their papers) on such approaches may be:

• Welcoming the Era of Deep Neuroevolution
• Accelerating Deep Neuroevolution: Train Atari in Hours on a Single Personal Computer

I didn't get around to reading them in detail yet, so I don't know exactly how similar they are, but certainly related.