0
$\begingroup$

I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. After training, the model always chooses the same action regardless of the state. For example (1, 2, 3) for all states. But I know this is wrong and not an optimal policy. But I cannot figure out why this is happening. The policy I am using is given below (code). Code is in Julia language and uses ReinforcementLearning.jl library.

# Now we use a QBasedPolicy and neural network to estimate values
# Create a flux based DNN for q - value estimation
STATE_SIZE = length(env.channels) # 3
ACTION_SIZE = length(action_set)    # 27     
model = Chain(
        Dense(STATE_SIZE, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, 48, relu),
        Dense(48, ACTION_SIZE)
    ) |> cpu

# optimizer 
η = 1f-2 # Learning rate  
η_decay = 1f-3
opt = Flux.Optimiser(ADAM(η), InvDecay(η_decay))

# Create policies for each agent
single_agent_policy = Agent(
        policy = QBasedPolicy(;
                learner = BasicDQNLearner(;
                    approximator = NeuralNetworkApproximator(;
                        model = model,
                        optimizer = opt
                    ),
                    min_replay_history = 500
                ),
                explorer = EpsilonGreedyExplorer(
                    kind = :linear,
                    ϵ_stable = 0,
                    ϵ_init = 0.5,
                    warmup_steps = 300,
                    decay_steps = 700,
                    is_training = true,
                    is_break_tie = false,
                    step = 1
                )
            ),
            trajectory = CircularArraySARTTrajectory(;
                        capacity = 500,
                        state=Array{Float64, 1} => (STATE_SIZE)
                    )
            )

During training, the model explores and exploits various actions in different states, but, during the testing/exploitation phase, it always outputs the same action for every state.

I searched for similar questions on the web, but none of the questions were well answered.

$\endgroup$
2
  • $\begingroup$ I am running in to the same problem now. Were you able to solve it? $\endgroup$ Jan 12, 2022 at 10:39
  • $\begingroup$ Unfortunately no :( $\endgroup$ Jan 14, 2022 at 12:34

0

You must log in to answer this question.

Browse other questions tagged .