Should the constraints reward also be normalized for all 10 constraints?
You should choose a "natural" balance between rewards where possible.
If you have many separate goals to take account of, ideally you should convert them all into some comparable metric that is meangful to the success of the agent. Such as a financial gain/loss, or energy ...
No. When a system where input and output is well known ie. the state space is fully known and defined using say physics (PDEs) then overfitting is desirable as there is no need to generalize. An ML model has low inference time compared to solving the PDEs or lookup table is of multiple terabytes.