I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses ReinforcementLearning.jl library.
# Now we use a QBasedPolicy and neural network to estimate values
# Create a flux based DNN for q - value estimation
STATE_SIZE = length(env.channels) # 3
ACTION_SIZE = length(action_set) # 27
model = Chain(
Dense(STATE_SIZE, 48, relu),
Dense(48, 48, relu),
Dense(48, 48, relu),
Dense(48, 48, relu),
Dense(48, ACTION_SIZE)
) |> cpu
# optimizer
η = 1f-2 # Learning rate
η_decay = 1f-3
opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))
# Create policies for each agent
single_agent_policy = Agent(
policy = QBasedPolicy(;
learner = BasicDQNLearner(;
approximator = NeuralNetworkApproximator(;
model = model,
optimizer = opt
),
min_replay_history = 500
),
explorer = EpsilonGreedyExplorer(
kind = :linear,
ϵ_stable = 0,
ϵ_init = 0.5,
warmup_steps = 300,
decay_steps = 700,
is_training = true,
is_break_tie = false,
step = 1
)
),
trajectory = CircularArraySARTTrajectory(;
capacity = 500,
state=Array{Float64, 1} => (STATE_SIZE)
)
)
During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.
I searched for similar questions on the web, but none of the questions were well answered.