Can agents be implemented with ML algorithms other than neural networks?

Question

Can agents be implemented with machine learning algorithms/models other than neural networks?

If so, how do I train an agent with some predefined rules? Can we use python programming for representing those rules?

Answer 1

Neural networks are not inherently part of reinforcement learning (a popular agent-based framework for describing control problems). In general, if you have an agent-based scenario, you are trying to optimise a function:

Policy( State ) -> Action

Where State can be any combination of current observations and history that seem relevant to the problem. The optimisation is usually over some measure of success, achieving goals etc. Reinforcement learning formalises these terms using Markov Decision Processes, which is a very general and successful approach, but not the only one.

Finding the best policy does not require neural networks. There are a lot of basic reinforcement learning algorithms that are defined without reference to NNs. For instance, Q-learning does not need neural networks, they are an optional extra. In addition, you do not need to even use RL - you can try to search for a suitable Policy function directly using a method such as Genetic Algorithms.

Focusing more on RL methods, the usual approach without any kind of function approximator is to just store an estimate of the value of each state or state/action pair. This value is a measure of long term reward, and has a formal definition in RL. Often called the tabular approach , because it is just a table of states and their estimated values, it works for small problems just fine. For example you can train an agent to play tic tac toe, find an optimal path that moves over a grid, or discover when to hold and when to twist in a simplified blackjack game.

However, problems start to occur with the most basic methods when the state space becomes large, which happens very easily. Tabular methods are OK up to a million or maybe ten million states (depending on how easy it is to take trial actions). These are enumerated states, so typically adding a single dimension to a problem multiplies the number of possible states by the number of options in that new dimension. So after a certain level of complexity, a simple table is not good enough, and you need a function approximator. Function approximators for RL ideally have the following properties:

• Can generalise from examples.

• Can be updated online as new data arrives, progressively forgetting old data.

• Can be differentiated with respect to their learning parameters.

Neural networks fit the bill. Some other popular supervised learning techniques do not - for instance Random Forests cannot usually be trained online.

In fact linear regression does work, and it is a very simple approach. For some problems, linear regression works very well in combination with reinforcement learning. You may need to do some careful feature engineering, and it is not always possible for more complex problems, but if you have a "medium-sized" agent-based problem - simpler than learning Go or playing video games - and you want to avoid using neural networks, then you should be able to attempt to train an agent using a combination of RL, such as Q-learning, plus linear regression.

Both Q-learning and linear regression have plenty of examples available online, and can be implemented from scratch using basic Python frameworks such as NumPy.