How to train the NN of simple agents given a reward system?

Question

I'm not an expert in AI or NN, I gathered most of the information I have from the internet, and I'm looking for advice and guidance.

I'm trying to design a NN that is going to be used by all the agents of my simulation (each agent will have its own matrix of weights). This is what I plan to have:

• The NN will have 1 input layer and 1 output layer (no hidden layers).
• The number of inputs will always be greater than the number of outputs.
• The outputs represent the probability of an action being taken by the agent (the output node with the highest value will identify the action that will be taken). Which means there are as many output nodes are there are actions.

When an agent takes an action it receives a reward: a number that represents how well the agent performed. This happens "online" that is, the agent is trained on the fly.

What I would like to know if how to best train the NN: that is, how to update the weights of my matrix to maximise the rewards long term.

From the research I made it seems this is close to the concept of Reinforcement Learning, but even if it was, it's not clear to me how to apply it to such a simple NN shape.

Answer 1

You might be able to glean what you want from Chapter 13 or Sutton & Barto's Reinforcement Learning: An Introduction, which deals with policy gradient algorithms, and includes pseudocode for a variety of agents based on linear approximation using softmax regression. From your description, you appear to be using - or should consider - softmax regression for the policy function.

Probably a good bet using your setup will be REINFORCE with baseline, which does not add requirements to track anything more than your current weights, sum of rewards seen after each state & action pair, and mean rewards (or other baseline). It does require an episodic environment though, such as a game that ends.

Essentially REINFORCE and its variants require you to figure out the return (sum of rewards received after each action to the end of the episode) and perform a gradient step update multiplied by that return (minus a baseline which can help with stability).

With REINFORCE and a discrete distribution, the gradient due to action selection is the same as supervised learning for multi-classification evaluated as if whichever action the agent took was the correct one. When you multiply this gradient by the return as well, it will make steps in the direction of better action choices larger. Regardless of the fact that you may train towards selecting many different actions, the better actions will win the race in the end due to this weighting, and have the highest probability of being selected after a large number of training episodes.

A multi-agent problem may bring you some problems, such as instability due to dependency of performance on what the other agents are doing. It is not possible to say how much of an issue this will be. However, I would advise not jumping straight into your main problem, and try some reinforcement learning methods with a single agent on toy problems first. If you build up to solving the problem you are currently facing, then you will be much more confident in how you have implemented the learning. There are plenty of stepping-stone projects and pre-built environments to solve if you search for them. A good starting point is OpenAI's Gym, which presents a standardised set of toy - and not so toy - problems that you can test any learning algorithm against.