How to structure the input data for non-vision deep reinforcement learning?

Question

I am currently designing a custom gym environment that is based on sensor data and I struggle a bit with structuring the data and designing the model. Virtually every resource I find online is kind of vision based, in the sense that it takes images as input arrays.

Data structure

For each time step, I get a couple of readings from each sensor. This would be an example reading from one sensor, with ... representing other variables.

{
  "temperature":"20.3",
  "humidity":"63",
  ...
}

Now I see two approaches how to structure the input array for a model (t being temperature, h being humidity):

[
 [t,h,...],
 [t,h,...],
]

[
 [t,t,...],
 [h,h,...],
]

Note that this represents the data for one time-step and is not representing a series of time-steps.

Question

Does it make a difference how I structure that data?

Answer 1

Does it make a difference how I structure that data?

Since your gym environment creates two values, t and h at one single step, I believe the way you structure your input matters.

Let's start with this one:

[
 [t,h,...],
 [t,h,...],
]

If I understand correctly, the index of the list is the input at the i-th step, so your input at step 0 is inputs = X[0] = [t,h,...]. This is the correct way. Your model sees t and h at the same time, which is what your environment produces.

If you structure your input in the second way:

[
 [t,t,...],
 [h,h,...],
]

This way, it means your model sees t first, then sees h in the next step. This is technically incorrect from what we assume, since each input should be independent of each other. What's worse, I see that your t and h are on different scales, so it is highly possible that the training will diverge.

However, depending on the difficulty of the task (which we don't know), you might or might not see a big difference between two structures. However, I recommend sticking with the first one.