This is a follow-up question to the answer to How can we use linear programming to solve an MDP?
Quick recap: the $max$ operators that appear in the Bellman optimality equations can be turned into a set of inequality constraints in hope of making a linear program out of the equation system.
As I understand, we want to minimize the value of every $V^*(s)$ independently. So, I would think we would solve as many linear programs as there are states. But the answer linked above rather say to minimize the sum $\sum_{s \in S} V^*(s)$.
I don't understand why this is correct. Is it a property of this very kind of problems? Is it a general property of linear programs?
