I want to build an AI that can convert an image of a subject into an anatomically accurate 3D model. To do this, I was thinking of adapting the following code for Deep Deterministic Policy Gradient: https://keras.io/examples/rl/ddpg_pendulum/

My reasons considering RL:

  1. I don't have the needed skill (3D modelling) to procure a large dataset for the project. I was hoping RL may help me overcome that by adapting to a smaller dataset through state-reward learning. I'm looking mainly for human and animal anatomical models. Those are hard to find in large numbers.

  2. My second concern is the required density (polygon count) of these models can be rather high. I'm not sure if it is computationally feasible for a NN to output high density models. However, I'm thinking an RL agent can step through and write each vertex one at a time in a 3D space. As compared to a single output layer in a feed-forward network.

However, that means it will have to handle a rather large state-space (an array of length 50,000 or higher).

With all of that said, RL has mainly been used in video games and simple control problems from the OpenAI gym. Is it a waste of time to use RL for this level of complexity?


1 Answer 1


Yes, RL is capable of learning complex functions, as it is a very general learning approach. However, if you have a direct goal of learning a complex function from example data, it will not really add anything to that process. Supervised learning will be more efficient.

RL doesn't directly address or have special mechanisms to deal with the two main issues you have:

  • Lack of training data

  • Multi-variable optimisation with a very large number of variables

Adding states, timesteps and trail-and-error learning to this problem does not make it more tractable. If the problem naturally presented itself as a sequence of simpler choices, then RL might help somewhat, but as far as I can see you have a very high dimensional function to learn, and there is no benefit from adding a layer of trial-and-error learning on top of it.

What you probably want is to find ways of constraining the problem, using domain knowledge and/or transfer learning from similar systems. For example, if you already know the types of creature in the photos being turned into meshes, you could probably start by categorising them in order to use some pre-defined meshes that you then adjust, as opposed to starting with figuring out the general plan of the mesh.

There are already systems that can turn pictures of human subjects into 3D models with approximately correct shape and pose. I would suggest you study those to understand how they work and whether the ideas in them can be re-used for your problem. Here is one called PIFuHD.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .