Variable Number of Inputs to Neural Networks

Question

So suppose that you have a real estate appraisal problem. You have some structured data, and some images exterior of home, bedrooms, kitchen, etc. The number of pictures taken is variable per observational unit, i.e. the house.

I understand the basics of combining an image processing neural net with tabular data for a single image. You chop off the final layer and feed in the embeddings of the image to your final model.

How would one deal with variable number of images? Where your unit of observation can have between zero and infinity images (theoretically no upper bound on number of images in observation)?

Answer 1

We can generalize both problem and solution by removing the specifics of housing.

Representing Forward Propagation

We have a function $f$ we wish to obtain via the training of an artificial network that produces a scalar result $s$, the sole dependent variable and the generalization of market price.

$s = f(s_1, s_2, ..., s_k, c_1, c_2, ..., c_v)$

The independent scalar variables $s_1$ through $s_k$ are the generalization of a constant number $k$ of property features from the tax authority, assessor's office, or inspection document. The question calls this structured data, however it is questionable whether $k$ is truly effectively constant. In normal practice, some of $s_i$ will be unassigned. Since the question overlooks the additional complexity of missing scalars, so will this answer.

The independent cube variables $c_1$ through $c_v$ are the generalizations of a variable number $v$ of property images from Google photography, real estate agents, buyers, sellers, and other potential sources. The dimensions of each cube are horizontal and vertical positions and pixel structure element number.

It is unlikely that the resolution values for each cube are uniform between samples in real life, which the question did not mention, so this answer will overlook that complexity and focus on the variability of $v$, the quantity of cubes representing images for a given example. Since $v$ cannot meaningfully be either negative or infinite, we can assume $v$ to be a non-negative integer.

Terminology

The observational unit should not be considered the house or the property but an image of it, which may be a member of a camera location and orientation category relative to the elements of the property. Each image capture is an observation, distinct in both Bergsonian and clock time. Each item under evaluation is an example from the sample of all items, in this case, all properties in the region for which prediction is attempted.

Design of a Solution

Each of the $v$ cubes demonstrate zero or more additional features of the item being evaluated, real properties in the question's specific case.

It may be reasonable to assume that such features may either positively or negative affect the example's label corresponding to the result $s$, but not both positively and negatively affect it. If that is the case, we can aggregate the features across the set of cubes for each example under the reasonable assumption that there are no points of inflection. Such may be reasonable because, for instance, a feature regarding the uniformity of paint coverage, lawn care, or roofing material may have no inflection point. Such aspects of the property cannot be too uniform. That makes the substitution straightforward.

A reasonably versatile way to generalize the aggregation of directionally consistent function is using a substitution, which may be what the question meant by feed in the embeddings of the image to your final model.

$s = f = f'(s_1, s_2, ..., s_k, v, h\big(v, \sum_{j = 1}^v g(c_j)\big)$

Note the elements of this substitution.

• $h$ is a vector function that normalizes the distribution of each feature found in the feature vector before using it as a set of inputs to $f'$.
• $g$ is a generalization of the cube, with the input (independent variable) being a cube representing the image and with the output (dependent variable) being a vector of features extracted.
• $v$, the number of cubes (visual observations) is fed along with extracted features into the function $f'$, which can be realized through the convergence of a multilayer perceptron.

If the images are grouped in terms of the location of the camera, then this principle can be applied iteratively, where $o$ is the number of distinct categories of camera orientation. In this case, the pairs $(v_z, h_z)$ represent the cube quantity and feature aggregations for image camera location category $z$.

$s = f = f'(s_1, s_2, ..., s_k, v_1, h_1, v_2, h_2, ... v_o, h_o)$

Given sensible models $h$ and $g$, training for prediction is straightforward.

Image feature extraction can be realized through ConvNet approaches such as OverFeat¹, AlexNet², CaffeNet³, GoogLeNet⁴, VGG 6⁵ or PatreoNet⁶. Tuning such models produces $g$.

The nature of function $h$ may be homogeneous or heterogeneous across dimensions. Each component of the feature vector arising from extraction can have applied to it a function such as any of these or others, where $q$ is the feature index, $p_{qi}$ is learning parameter i.

1. $h_q(x) = x$
2. $h_q(x) = \large{x^{p_{q1} + p_{q2} v}}$
3. $h_q(x) = \log (x + p_{q1} v + p_{q2})$
4. $h_q(x) = \large{\epsilon^x}$
5. Others

Scalar function 1 is best when the designer wishes the convergence during the training of $f'$ in such a way that normalization is accomplished in the net. It is a good choice for features where its frequency of occurrence and magnitude are across the entire set of images for a given example item is roughly proportional to the resulting value of that item.

Function 2 presents flexibility in normalization curvature with respect to the number of cubes. Function 3 presents attenuation of the frequency of feature occurrence. Function 4 presents compounding of feature effect with recurrence in the images of the same example.

The key is then the selection of how to deal with the substitution in the training in terms of procedure and wiring of corrective signaling. Procedurally, there are three options.

• Train both the ConvNet function $g$ and the multilayer perceptron function $f'$ together, extending the applicable principles of back-propagation and gradient descent.
• Extract features first, tuning the ConvNet corresponding to $g$ prior to training the network corresponding to $f'$. The advantage of this approach is manual control over and interim evaluation of feature extraction.
• Use something similar to the mini-batch approach to find a balance between the above two extremes.

———

Footnotes

[1] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229v4

[2] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Neural Information Processing Systems, 2012, pp. 1106–1114

[3] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093

[4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, arXiv preprint arXiv:1409.4842

[5] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556

[6] K. Nogueira, W. O. Miranda, J. A. Dos Santos, Improving spatial feature representation from aerial scenes by using convolutional networks, in: Graphics, Patterns and Images (SIBGRAPI), 2015 28th SIBGRAPI Conference on, IEEE, 2015, pp. 289–296.

Answer 2

I can think of 4 options:

1. One option is to divide the data so that each data point only has one picture but we have multiple data points per one real estate with structured data duplicated. Then calculate the forecasts and average the resulting predicted price between data points belonging to one real estate. Here we probably assume that the quality of pictures is more important than what is being shot.

2. The other option is to divide the pictures to categories: bedroom1, bathroom, bedroom2, kitchen and so on. Then for the missing pictures use a black square. This way you will be able to have multiple pictures in one data point.

3. The third option would be to have places for the maximum amount of pictures and fill only the first pictures that are available. And the other pictures fill with black squares/Nans.

4. The best option would probably be combining option number two and three. So you have categories and maximum number of pictures per category. So you will have four categories for bathroom1 if the maximum is 4 pictures of the first bathroom:

   Bathroom1_1, Bathroom1_2, Bathroom1_3, Bathroom1_4, Bedroom1_1, Bedroom1_2, Bedroom1_3, Bedroom2_1, Bedroom2_2 ... and so on.

The gradient boosting algorithms are very good at working with missing values. So probably lightgbm and xgboost as last layers could give good results.

So in the end you can check all the variants and even do ensembles from the variants that give the best results.