What components of reinforcement learning influence the result the most?

Question

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it:

• Formalising the agent environment (like the design of state-, action-space and reward-structure)
• Selection of learning algorithm
• Selection of network architecture and size
• Design of the training setup

It is an agent in a 3D environment with simulated physics (in Unity), the domain being a real-time strategical game. It is an environment with constraint training data, so sample efficiency is very important.

Now my question: I do anticipate that the design of the state- and action space will have a big impact on the training result, especially in this environment with little training data.

However, is there a way one can clearly prioritise what components will be the most important ones for an RL setting?

Time is limited, and, for me, as a beginner, it seems to be quite difficult to determine what component will be the most important one and needs the most focus. Testing only the hyper-parameters of a learning algorithm thoroughly will take in itself a long time. And obviously disregarding any component will result in bad results.

Is there a way to know on which component one should focus more?

Answer 1

I don't think there's a strategy that applies to all cases. In some cases, the reward function may need to be carefully designed (e.g. a self-driving car), but, in other cases, the reward function might be quickly designed (e.g. chess) and other parts of the RL system may require more care.

Here you can find tips to approach an RL problem. For example, one tip that I find useful, although probably obvious once you hear of it, is to compare your policy with a random policy. See also these tips.