On creating custom environments:
... always normalize your observation space when you can, i.e., when you know the boundaries
You could normalize them as part of the environment's state space or before passing them as input to the policy. Depending on the the agent's algorithm implementation, what works for you may vary.
(See this ...