If the AI alignment problem is one of the most pressing issues of our time, could AI itself augment our (i.e., human) quest to solve the alignment problem? Or would AI itself actually be counter-productive for such a meta-level goal?
Yes, this is one the most popular approaches today. For instance a narrow AI model is built to capture the preference of the user and then this model is used to dispense reward to a reinforcement learning agent.
The narrow AI model is often called Reward Model or Preference Model.
The recent RLHF training technique used for chatGPT uses this approach.
This can be automated further: https://arxiv.org/abs/2212.08073