0
$\begingroup$

I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to make a basic blueprint for myself and the community to use. I've put in a lot of work and I feel like it should basically be complete, but I can't find out why the agent keeps (very quickly) converging on a single action selection (even though it's being penalized for doing so, even testing at 300 iterations). I checked the loss variables and they seem to be working as expected. Although I'm great with JavaScript and have a solid understanding of how everything works, I'm not a professional with Python or TensorFlow.js (yet).

I need some people with experience in the following areas to review my code: Reinforcement Learning, TD3 (or DDPG), Tensorflow.JS.

Everything is on GitHub at https://github.com/CloudZero2049/TD3-TensorFlowJS

All the info about the project is in the README. TD3script.js is the only one to worry about, there's also one without comments.

I have been reviewing the code and math extensively everything seems to be working properly. But when testing the agent quickly settles on a single bad action even though it's being penalized. I tried 300 iterations, 200 warmup, 100 batch_size to see if it was a time issue without a change. I feel confident in the structure and math but something is clearly wrong. After going over the code so many times and making improvements I've finally hit the wall, I don't see how I can proceed.

I expected at some point for the agent to realize it was doing something wrong and try something new. I would have gotten much better results if the actions were completely random.

Improve this question
$\endgroup$
2
  • $\begingroup$ One thing to check is whether you are normalising the NN inputs, it's a common omission for beginners. Unlikely anyone is going to review your project code for you, I suggest you reach out to the community you are hoping to serve with the js conversion. $\endgroup$
    – Neil Slater
    Commented Nov 11, 2023 at 10:44
  • $\begingroup$ I'm trying this now, thank you. I learned this early on and completely forgot. Now I'm not sure if it's better to normalize states and rewards before or after they are sent to the memory buffer. I'm trying "before" first because it's much easier to do. $\endgroup$
    – CloudZero
    Commented Nov 11, 2023 at 21:04

1 Answer 1

Reset to default
0
$\begingroup$

The primary issue I was having was that I wasn't normalizing the input data before sending it through the system. I can confidently say that it is working now.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .