In Reinforcement Learning (RL), a reward function is part of the problem definition and should:
Be based primarily on the goals of the agent.
Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should not use other data than those four things, but you also do not have to use any of them. This is important, as using any other data stops your environment from being a Markov Decision Process (MDP).
Given the first point, as direct and simple as possible. In many situations this is all you need. A reward of +1 for winning a game, 0 for a draw and -1 for losing is enough to fully define the goals of most 2-player games.
In general, have positive rewards for things you want the agent to achieve or repeat, and negative rewards for things you want the agent to avoid or minimise doing. It is common for instance to have a fixed reward of -1 per time step when the objective is to complete a task as fast as possible.
In general, reward 0 for anything which is not directly related to the goals. This allows the agent to learn for itself whether a trajectory that uses particular states/actions or time resources is worthwhile or not.
Be scaled for convenience. Scaling all rewards by a common factor does not matter at a theoretical level, as the exact same behaviour will be optimal. In practice you want the sums of reward to be easy to assess by yourself as you analyse results of learning, and you also want whatever technical solution you implement to be able to cope with the range of values. Simple numbers such as +1/-1 achieve that for basic rewards.
Ideally, you should avoid using heuristic functions that reward an agent for interim goals or results, as that inserts your opinion about how the problem should be solved into the system, and may not in fact be optimal given the goals. In fact you can view the purpose of value-based RL is learning a good heuristic function (the value function) from the more sparse reward function. If you already had a good heuristic function then you may not need RL at all.
You may need to compare very different parts of the outcome in a single reward function. This can be hard to balance correctly, as the reward function is a single scalar value and you have to define what it means to balance between different objectives within a single scenario. If you do have very different metrics that you want to combine then you need to think harder about what that means:
Where possible, flatten the reward signal into the same units and base your goals around them. For instance, in business and production processes if may be possible to use currency as the units of reward and convert things such as energy used, transport distance, etc., into that currency.
For highly negative/unwanted outcomes, instead of assigning a negative reward, consider whether a constraint on the environment is more appropriate.
You may have opinions about valid solutions to the environment that you want the agent to use. In which case you can extend or modify the system of rewards to reflect that - e.g. provide a reward for achieving some interim sub-goal, even if it is not directly a result that you care about. This is called reward shaping, and can help in practical ways in difficult problems, but you have to take extra care not to break things.
There are also more sophisticated approaches that use multiple value schemes or no externally applied ones, such as hierarchical reinforcement learning or intrinsic rewards. These may be necessary to address more complex "real life" environments, but are still subject of active research. So bear in mind that all the above advice describes the current mainstream of RL, and there are more options the deeper you research the topic.
Is this purely experimental and down to the programmer of the environment. So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?
Generally no. You should base the reward function on the analysis of the problem and your learning goals. And this should be done at the start, before experimenting with hyper-parameters which define the learning process.
If you are trying different values, especially different relative values between different aspects of a problem, then you may be changing what it means for the agent to behave optimally. That might be what you want to do, because you are looking at how you want to frame the original problem to achieve a specific behaviour.
However, outside of inverse reinforcement learning, it is more usual to want an optimal solution to a well-defined problem, as opposed to a solution that matches some other observation that you are willing to change the problem definition to meet.
So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?
This is usually not the case.
Instead, think about how you want to define the goals of the agent. Write reward functions that encapsulate those goals. Then focus on changes to the agent that allow it to better learn how to achieve those goals.
Now, you can do it the way round, as you suggest. But what you are doing, in that case, is changing the problem definition, and seeing how well a certain kind of agent can cope with solving each kind of problem.
