Machine learning has been used to automatically learn new optimization/learning algorithms. This task is often known as meta-learning, i.e. you learn to learn, in this case, an optimization algorithm, but note that meta-learning does not just refer to learning optimization algorithms (see this blog post).
The blog post Learning to Optimize with Reinforcement Learning (2017) is a good introduction to the topic and focuses on the approach proposed in this paper Learning to Optimize (2016), which uses reinforcement learning to solve this problem: more specifically, they learn a policy (in practice, represented as a neural network) that represents the learned optimization algorithm.
There are other related approaches: for example, you may be interested in the paper Learning to learn by gradient descent by gradient descent (2016, NeurIPS).