3
$\begingroup$

Hello :) I'm pretty new to this community, so let me know if I posted anything incorrectly and I'll try to change it.

I'm working on the project which aim is to create self-driving agent in CARLA. I built a neural network Xception (decaying ε-greedy). The other parameters are:

EPISODES: 100
GAMMA: 0.3
EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001 BATCH: 16

Due to the limited computer resources I chose 100 or 300 epochs to train the model, but it generates much fluctuations: enter image description here enter image description here

EPISODES: 100
GAMMA: 0.7 EPSILON_DECAY: 0.9
MIN_EPSILON: 0.001 BATCH: 16

enter image description here Can anyone suggest how can I improve my results? Or it is only the issue of small number of epochs?

Improve this question
$\endgroup$
  • $\begingroup$ how come your gamma is 0.3? This seems quite low, and would suggest you care more about immediate returns than future returns (not something that is typically the case in RL). $\endgroup$ – David Ireland Oct 20 '20 at 10:54
  • $\begingroup$ Changing otherwise to gamma = 0.7 didn't help with fluctuations. I'm not sure if gamma is the issue. Maybe it is connected with the small amount of epochs? $\endgroup$ – martin Oct 20 '20 at 15:01
  • $\begingroup$ A more common value for gamma would be 0.9 or 0.99. I would try those. $\endgroup$ – S2673 Oct 20 '20 at 15:34
  • $\begingroup$ @S2673 I'm not sure if this would be ok with autonomous vehicle... $\endgroup$ – martin Oct 20 '20 at 15:47
  • $\begingroup$ What do you mean? Do you think it would be dangerous if it was willing to crash as long as it got somewhere? I would still try it anyway to see if the agent learns and then you can change gamma or the reward system. $\endgroup$ – S2673 Oct 20 '20 at 15:49
1
$\begingroup$

It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.

Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling

Improve this answer
$\endgroup$
  • $\begingroup$ But are those fluctuations in my example a kind of unnormal thing? Or are they acceptable for someone not very experienced in field of deep learning? $\endgroup$ – martin Oct 20 '20 at 15:08
  • 1
    $\begingroup$ in my experience, fluctuations is common $\endgroup$ – Aray Karjauv Oct 20 '20 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.