0
$\begingroup$

Context: I want to build a DQN with as CNN for calculating its Q value on each step.

Enviroment's status can be described by the attributes of 3 machines (each one with own attributes). I'd also like to add some history of recent readings so DQN can understand what is happenening and not only a picture of current readings.

Let's say machine A, B and C have attributes {a1, a2, a3}, {b1, b2, b3} and {c1, c2}

I want to pass the last n reads. So CNN input should have something like:

[
    [[a_11, a_12, a_13], [b_11, b_12, b_13], [c_11, c_12]],
    [[a_21, a_22, a_23], [b_21, b_22, b_23], [c_21, c_22]],
    ...
    [[a_n1, a_n2, a_n3], [b_n1, b_n2, b_n3], [c_n1, c_n2]]
]

Problem: I can't figure out how to represent this in a matrix because different machines have different amount of attributes

My thoughts:

  • Each machine is a 1 dimension vector
  • 3 machines can be interpreted in a 2D CNN. But not all machines are the same length...
  • As 2D does not work, 3D CNN by adding time will not work as well

Suboptimal solution idea:

  • Flatten all machine attributes to a single read as [a1, a2, a3, b1, ..., c2]
  • Make it 2D with time
  • Use 2D CNN with convolusional operations work in columns (not between them).

Questions:

  1. Any better way to organize CNN input?
  2. Maybe other kind of NN to estimate Q-Values is a better fit in this case?
$\endgroup$

1 Answer 1

2
$\begingroup$

CNNs have an inductive bias of locality. They work great on images because squares of inputs often are of an object. It is a hint to the ML algorithm that very far away pixels should not be considered. I can't confirm that CNN is wrong here but unless there is some locality to these machine attributes it may be more work than it's worth.

I would likely keep it dense, the values all get considered against each other in a dense layer anyway. To consider history it is then natural to convolute with history or LSTM, GRU, etc.

It is of course possible to try and force locality by grouping inputs into multiple unconnected dense networks and then merging them at a later layer. Hinting can sometimes help but unless one knows of something specific it may be best to leave it simple.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .