0
$\begingroup$

I'm making a simple deep Q learning algorithm, with cartpole-v1 env.

Like you can see in chart, after many episodes the reward decrease, some possible reasons?

The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)

The neural network is made from scratch the complete code is here: https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project

# %%
from core.dqn_agent import DQNAgent
from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
from cartpole.cartpole_wrapper import CartPoleWrapper
import gym
import numpy as np
import torch
from tqdm import tqdm
import glob
import os
from IPython.display import Video
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from itertools import cycle
import sys
import shutil
from pathlib import Path
import shutil
import pyvirtualdisplay
_display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
_ = _display.start()



# %% [markdown]
# Initialize deep Q-learning agent, neural network, and parameters
# %%
np.random.seed(20)
agent = DQNAgent(env=CartPoleWrapper(gym.make("CartPole-v1")),
                nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)

DISCOUNT_FACTOR = 0.995
LEARNING_RATE = 0.0001

n_episodes = []
total_rewards = []
number_steps = []
total_episodes = 0


# %% [markdown]
# Training
# %%
while total_episodes <= 10000:
    total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
    print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
    n_episodes.append(total_episodes)
    total_rewards.append(total_reward)
    number_steps.append(steps)

    for i in tqdm(range(50), 'learning...'):
        agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
    total_episodes += i+1

chart

$\endgroup$
4
  • $\begingroup$ Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help. $\endgroup$
    – S2673
    May 11 at 19:57
  • $\begingroup$ Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why $\endgroup$ May 12 at 7:03
  • $\begingroup$ Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case. $\endgroup$
    – ddaedalus
    May 27 at 0:09
  • $\begingroup$ batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay, or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint! $\endgroup$ May 27 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.