How to structure the input data for non-vision deep reinforcement learning?

I am currently designing a custom gym environment that is based on sensor data and I struggle a bit with structuring the data and designing the model. Virtually every resource I find online is kind of vision based, in the sense that it takes images as input arrays.

Data structure

For each time step, I get a couple of readings from each sensor. This would be an example reading from one sensor, with ... representing other variables.

{
"temperature":"20.3",
"humidity":"63",
...
}


Now I see two approaches how to structure the input array for a model (t being temperature, h being humidity):

[
[t,h,...],
[t,h,...],
]

[
[t,t,...],
[h,h,...],
]


Note that this represents the data for one time-step and is not representing a series of time-steps.

Question

Does it make a difference how I structure that data?

• I'm gonna mark this for later because I think I can help, but it's late here Commented Feb 26, 2023 at 17:37
• Hi @94621 and welcome to AI Stack Exchange! This stack exchange website seems to be very particular about asking a single question per post. If possible, please condense your questions into a single question. I know that it might be difficult, but this post will most likely be closed if action isn't taken. Thank you for posting, and we hope to see more of your questions on this site! Commented Mar 2, 2023 at 18:11
• Thanks for the hint @DeepQZero. I will edit the question accordingly Commented Mar 3, 2023 at 13:28

Does it make a difference how I structure that data?

Since your gym environment creates two values, t and h at one single step, I believe the way you structure your input matters.

[
[t,h,...],
[t,h,...],
]


If I understand correctly, the index of the list is the input at the i-th step, so your input at step 0 is inputs = X[0] = [t,h,...]. This is the correct way. Your model sees t and h at the same time, which is what your environment produces.

If you structure your input in the second way:

[
[t,t,...],
[h,h,...],
]


This way, it means your model sees t first, then sees h in the next step. This is technically incorrect from what we assume, since each input should be independent of each other. What's worse, I see that your t and h are on different scales, so it is highly possible that the training will diverge.

However, depending on the difficulty of the task (which we don't know), you might or might not see a big difference between two structures. However, I recommend sticking with the first one.

• Thanks for your answer @minh-long-luu. I think my description is a bit ambiguous: The list I am describing is actually for one time-step, meaning that I have readings from multiple sensors per step. I will edit my question to make that more explicit. Commented Mar 6, 2023 at 8:01