0
$\begingroup$

Is it possible to train a DL model that will generate a full resolution 2D image based on few numbers describing this image and what type of model or architecture would that be? enter image description here

What I want to achieve is that I deliver to the model some numbers for example describing positions of objects on the screen and number describing how lit the scene is and I get back a 2D image with objects in their correct positions and proper lighting, but for one set of input data values I will get always one same image (see image above). These input data also could be anything else than positions and lighting, these are only examples helping to visualize what I mean.

This all, of course, assuming that I have a lot of annotated training data that consists of images and labels of the objects' positions and scene lighting values.

EDIT: The final model would be trained on real images taken from Full HD camera, not some simple shapes like presented here, that I did only to explain better my question.

Improve this question
$\endgroup$
4
  • $\begingroup$ I do not see the reason to refer to any kind of deep learning for this problem. You have objects and properties of the scene and just render the scene and output view from a certain angle. $\endgroup$
    – spiridon_the_sun_rotator
    Jul 23, 2021 at 9:08
  • $\begingroup$ @spiridon_the_sun_rotator Well, the reason is that this is only simplification of my problem that I done to help visualise what I am asking for. Later, I want to train suggested model on data of real images taken with Full HD camera and make sort of a realistic scene renderer, generated by DL. $\endgroup$
    – GKozinski
    Jul 23, 2021 at 11:01
  • $\begingroup$ Also, I don't see why, for experimental/research reasons, I should restrain myself from using other methods of doing tasks that can be done with regular/simple methods. My question was about whether such DL models exist, not how to render scene. $\endgroup$
    – GKozinski
    Jul 23, 2021 at 11:05
  • $\begingroup$ Did you take a look at GANs? $\endgroup$
    – Aray Karjauv
    Jul 23, 2021 at 18:00

1 Answer 1

Reset to default
0
$\begingroup$

Probably closely related to the problem of interest would be something akin Neural Radiance Fields (NeRF for short) https://www.matthewtancik.com/nerf.

The model takes several images from different angles and view of the scene and learns a 3d representation of the scene, that can be used to sample novel views of this scene (not present in the training data).

enter image description here

The knowledge of the scene is encoded in the weights of MLP. Namely, the renderer is a function that takes coordinates $(x, y, z)$ and viewing angle $(\theta, \phi)$ as an input, and outputs the color (RGB) and the density $\sigma$.

From

For your problem, one can modify, the network, that is takes in addition to the coordinates and angles, other properties of the scene, such as lightning (some scalar), e.t.c and outputs again (RGB) vector and density $\sigma$.

In order for procedure to be successful, I would expect, that model needs data with different lighting, and provided there are enough data points, one could synthesize the view with given lightning.

The presence of multiple objects can be probably taken into account, by passing a set of coordinates $(x_i, y_i, z_i)$ - but this seems to complicate things a lot, and the training procedure.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is very close to what I need indeed, with the different output, I think their model outputs some data like points in cloud, and only later they make a scene from this by use of traditional rendering methods. What I want to do is to make such a renderer to produce ready 2D images directly from input data. But I think I will try something like reversed FC network or reversed CNN network, that instead of making predictions from images it makes images from predictions, sort of thing. $\endgroup$
    – GKozinski
    Jul 24, 2021 at 6:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .