Can residual connections be beneficial when we have a small training dataset?
The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
It is very hard to tell, a priori, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.
In this case, you are wondering whether residual networks using skip connections might help when you have a relatively low amount of training data.
On the pro side, effects of skip connections that help correct vanishing gradients, and treat each block as learning the difference from an identity function, will still work for your problem. That means you will have some freedom to explore adding layers without worrying about the negative impacts of doing so.
On the con side, it is unlikley that you will benefit from very deep networks as there will not be enough examples to learn truly complex functions from.
You may find that adding depth, but reducing "width", i.e. the number of artificial neurons in each layer, will work.
If you have a low amount of training data and a difficult problem to solve, then residual networks are not a magic fix. The best you could hope for is a relatively stable statistical model that works on the simpler differences between your training examples. However, it may be possible that tuning a neural network by searching through different architectures will be a worthwhile exercise.
I would also suggest that for one of the good networks that you vary the number of examples used to train the network in multiple training runs, and plot a learning graph showing number of examples versus accuracy (or other metric that you may be interested in). This graph will help you decide whether collecting more training data would be worthwhile because you will have a rough estimate of the gradient for how much new training examples could improve your results.
