The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally importantly forgetting older data. These algorithms learn from fixed datasets, and must be re-trained with whole new datasets if the target dataset changes, in scenarios also known as non-stationary problems.
This is a major issue in reinforcement learning for control problems (when you want to find an optimal policy). RL control problems always generate non-stationary target data, because they maintain a current policy - or for off-policy approches, current target policy. That current policy then changes over time as the agent improves its estimates (of e.g. the Q table), which means it is necessary to forget older training data and replace with newer.
So one requirement in RL is for online learning algorithms that can handle non-stationary data well - both incorporating new data and smoothly forgetting older data. That includes neural networks. There are not many other non-linear approximators that can do so simply.
It is possible to compensate for this lack of online learning in various ways. Some of the model classes, such as random forests and support vector machines, can be extended to learn online - although this seems not to be explored much in deep RL, perhaps due to complexity, or limitations that make it still less flexible choice than neural networks (opinion: looking at the recent dates on these papers compared to the longer history of using NNs with RL, there may be an element of intertia too, and possibility for revisiting using these models in a RL context).
You could modify the outer algorithm to be more like policy iteration, collecting large amounts of data for the current policy to form a large batch that is then learned from to replace the previous approximate value function and in turn update the policy from that. That would work for most approximators in theory, but may be very slow and introduces hyperparameter of batch size that will have a large impact.
There are unwanted side-effect to online learning in RL using approximators, such as the risk of catastrophic forgetting, where a well-trained agent is not able to distinguish bad choices from good ones because it has not experienced a bad decision for many time steps, and has generalised that "everything is good now". This would not be fixed in RL by using large batches and non-online algorithms - if anything it could be worse in those cases, because they would effectively be forced to forget everything between batches. The usual fix for catastrophic forgetting in RL is adding some management of the dataset - e.g. keeping some amount of older experience even though it is technically not part of the current policy's generated data.
they also have a huge number of trainable parameters and are extremely prone to overfitting
Overfitting of approximators can happen, but may be less of an issue in practice with RL compared to smaller fixed datasets, because agents are continuously and actively generating new data. Complex environments with lots of resources used to generate experience behave in a lot of ways like "big data" datasets, playing to the strengths of large flexible models.
In RL there is commonly an overlap with processing image data (and possibly other sensors with return data that can be structured into some grid or graph). In the problem domains related to image comprehension, neural networks are state of the art, so it is no surprise to see them used for approximating action values in the orginal DQN paper for example - with the right architecture NNs are measurably superior, in terms of error metrics, to SVMs, RFs, boosting et al, when fitting this kind of sensor data.
