From what I can find, reinforcement algorithms work on a grid or 2-dimensional environment.
A lot of teaching materials use a "grid world" presentation to demonstrate basic reinforcement learning (RL). However, the underlying Markov Decision Process (MDP) theory works on an arbitrary graph of connected states. This graph could be based on subdividing a metric space of any dimensions into a grid of the same dimensions (and using tiles of any shape that worked in that dimension). However, it is not limited to that, the state space does not need to be a metric that represents distances or physical properties.
In practice, the set of states can be arbitrary objects, connected via state transitions in any consistent way. Provided the transition probability function $p(s'|s,a)$ is consistent, the environment could be used in a RL problem.
A very common state description is that the state is a vector of numbers that capture all the variables relevant to the problem. The environment can then be measurements taken in the real world of those variables, or the same quantities provided by a simulation. That state vector can be of any size, and have arbitrary constraints on individual components. This is no different from numerical representations of other machine learning problems, such as the inputs allowed to a neural network.
The environment would be a lighthouse beam, the state would be the brightness seen at the sensor for a given orientation, and the agent would be the approximate brightness/orientation?
Something not quite right about the description there. There does not seem to be any action that the agent takes.
What would the reward be?
It would be whatever measure of reaching a goal or maintaining a "good" result that is appropriate for the problem. You do not give any information about the goal in your description.
If your goal is to light up a moving sensor with the highest brightness, then the brightness measured at the sensor, transformed into suitable units, would seem to be a good candidate for a reward function (you would also need the state to give information about the target - where it had been seen last for instance). Assuming the problem is continuous, you would also need a discount factor.
What reinforcement learning algorithm would I use to approximate the lighthouse orientation given sensor brightnesses?
Generally RL algorithms estimate rewards, or generate policies. If the lighthouse orientation is the action you wanted to take, then pretty much all RL algorithms can do one or the other to allow you to do this. The differences are in things like complexity or speed of the algorithm, what approximations you are willing to take etc.
You don't give enough information about the problem to even nearly suggest a "best" algorithm. Before you start, you will need to determine a more thorough description of state, action and rewards, that will define the problem. Once you have a more formal description of the problem, that may suggest which algorithms would be good starting points.