5
$\begingroup$

A dataset is given which contains textual data (year, number of rooms, location) and visual data (an jpeg image of the house). The neural network has the task to predict the price of the property. As an example, the training dataset consists of some values of a computer game simulation (a city simulation) and the aim is to determine the housing price for new unseen real estate.

The problem is, that that the number of pictures in the input dataset is fluctuating. Sometimes no images are given and sometimes more. That means the image recognition engine must form a sublayer in the overall neural network. It is some kind of aggregation problem to transform first visual data into a textual description of the image and aggregate it then with the other textual information.

original message:

So suppose that you have a real estate appraisal problem. You have some structured data, and some images exterior of home, bedrooms, kitchen, etc. The number of pictures taken is variable per observational unit, i.e. the house.

I understand the basics of combining an image processing neural net with tabular data for a single image. You chop off the final layer and feed in the embeddings of the image to your final model.

How would one deal with variable number of images? Where your unit of observation can have between zero and infinity images (theoretically no upper bound on number of images in observation)?

$\endgroup$
  • 1
    $\begingroup$ I'm not sure the edit was necessary. In fact the computer game simulation bit is really confusing to me, since I'm asking about real world MLS data. And I think that you meant tabular data instead of textual. As textual seems to imply that we are talking about something like realtor description of the property. $\endgroup$ – Ryan Aug 23 '18 at 15:10
  • $\begingroup$ It's only a suggestion, you can skip it if you want. It helps the newbies to become familiar with the edit button. $\endgroup$ – Manuel Rodriguez Aug 25 '18 at 17:10
  • $\begingroup$ @ManuelRodriguez, the edit is less clear and distorted with respect to the original question. $\endgroup$ – FauChristian Aug 31 '18 at 9:27
  • $\begingroup$ @FauChristian It has probably to do with the blank slots. The idea is to use placeholder pictures which can be filled or not. $\endgroup$ – Manuel Rodriguez Aug 31 '18 at 10:21
1
$\begingroup$

We can generalize both problem and solution by removing the specifics of housing.

Representing Forward Propagation

We have a function $f$ we wish to obtain via the training of an artificial network that produces a scalar result $s$, the sole dependent variable and the generalization of market price.

$s = f(s_1, s_2, ..., s_k, c_1, c_2, ..., c_v)$

The independent scalar variables $s_1$ through $s_k$ are the generalization of a constant number $k$ of property features from the tax authority, assessor's office, or inspection document. The question calls this structured data, however it is questionable whether $k$ is truly effectively constant. In normal practice, some of $s_i$ will be unassigned. Since the question overlooks the additional complexity of missing scalars, so will this answer.

The independent cube variables $c_1$ through $c_v$ are the generalizations of a variable number $v$ of property images from Google photography, real estate agents, buyers, sellers, and other potential sources. The dimensions of each cube are horizontal and vertical positions and pixel structure element number.

It is unlikely that the resolution values for each cube are uniform between samples in real life, which the question did not mention, so this answer will overlook that complexity and focus on the variability of $v$, the quantity of cubes representing images for a given example. Since $v$ cannot meaningfully be either negative or infinite, we can assume $v$ to be a non-negative integer.

Terminology

The observational unit should not be considered the house or the property but an image of it, which may be a member of a camera location and orientation category relative to the elements of the property. Each image capture is an observation, distinct in both Bergsonian and clock time. Each item under evaluation is an example from the sample of all items, in this case, all properties in the region for which prediction is attempted.

Design of a Solution

Each of the $v$ cubes demonstrate zero or more additional features of the item being evaluated, real properties in the question's specific case.

It may be reasonable to assume that such features may either positively or negative affect the example's label corresponding to the result $s$, but not both positively and negatively affect it. If that is the case, we can aggregate the features across the set of cubes for each example under the reasonable assumption that there are no points of inflection. Such may be reasonable because, for instance, a feature regarding the uniformity of paint coverage, lawn care, or roofing material may have no inflection point. Such aspects of the property cannot be too uniform. That makes the substitution straightforward.

A reasonably versatile way to generalize the aggregation of directionally consistent function is using a substitution, which may be what the question meant by feed in the embeddings of the image to your final model.

$s = f = f'(s_1, s_2, ..., s_k, v, h\big(v, \sum_{j = 1}^v g(c_j)\big)$

Note the elements of this substitution.

  • $h$ is a vector function that normalizes the distribution of each feature found in the feature vector before using it as a set of inputs to $f'$.
  • $g$ is a generalization of the cube, with the input (independent variable) being a cube representing the image and with the output (dependent variable) being a vector of features extracted.
  • $v$, the number of cubes (visual observations) is fed along with extracted features into the function $f'$, which can be realized through the convergence of a multilayer perceptron.

If the images are grouped in terms of the location of the camera, then this principle can be applied iteratively, where $o$ is the number of distinct categories of camera orientation. In this case, the pairs $(v_z, h_z)$ represent the cube quantity and feature aggregations for image camera location category $z$.

$s = f = f'(s_1, s_2, ..., s_k, v_1, h_1, v_2, h_2, ... v_o, h_o)$

Given sensible models $h$ and $g$, training for prediction is straightforward.

Image feature extraction can be realized through ConvNet approaches such as OverFeat1, AlexNet2, CaffeNet3, GoogLeNet4, VGG 65 or PatreoNet6. Tuning such models produces $g$.

The nature of function $h$ may be homogeneous or heterogeneous across dimensions. Each component of the feature vector arising from extraction can have applied to it a function such as any of these or others, where $q$ is the feature index, $p_{qi}$ is learning parameter i.

  1. $h_q(x) = x$
  2. $h_q(x) = \large{x^{p_{q1} + p_{q2} v}}$
  3. $h_q(x) = \log (x + p_{q1} v + p_{q2})$
  4. $h_q(x) = \large{\epsilon^x}$
  5. Others

Scalar function 1 is best when the designer wishes the convergence during the training of $f'$ in such a way that normalization is accomplished in the net. It is a good choice for features where its frequency of occurrence and magnitude are across the entire set of images for a given example item is roughly proportional to the resulting value of that item.

Function 2 presents flexibility in normalization curvature with respect to the number of cubes. Function 3 presents attenuation of the frequency of feature occurrence. Function 4 presents compounding of feature effect with recurrence in the images of the same example.

The key is then the selection of how to deal with the substitution in the training in terms of procedure and wiring of corrective signaling. Procedurally, there are three options.

  • Train both the ConvNet function $g$ and the multilayer perceptron function $f'$ together, extending the applicable principles of back-propagation and gradient descent.
  • Extract features first, tuning the ConvNet corresponding to $g$ prior to training the network corresponding to $f'$. The advantage of this approach is manual control over and interim evaluation of feature extraction.
  • Use something similar to the mini-batch approach to find a balance between the above two extremes.

———

Footnotes

[1] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229v4

[2] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Neural Information Processing Systems, 2012, pp. 1106–1114

[3] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093

[4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, arXiv preprint arXiv:1409.4842

[5] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556

[6] K. Nogueira, W. O. Miranda, J. A. Dos Santos, Improving spatial feature representation from aerial scenes by using convolutional networks, in: Graphics, Patterns and Images (SIBGRAPI), 2015 28th SIBGRAPI Conference on, IEEE, 2015, pp. 289–296.

$\endgroup$
  • $\begingroup$ I like this answer a lot, you are quite clear to the point of being academic. I have one question though for clarification. In the function h(v,∑vj=1g(cj), are you saying to simply add the feature vectors from each image together and apply a normalization scheme to bring them back to a reasonable numerical range? Or is the normalization performing some other function here? $\endgroup$ – Ryan Aug 31 '18 at 15:00
  • $\begingroup$ The index j is the index on the cube array (set of images for a single property), not the index of the feature extracted from the images. Let's say green lawns sell houses, an automatically extracted feature might be called lawn greenness upon introspection, and if there are three images the summation would add the greenness of them. Zero images would have a greenness of zero, which is fine because v is an input parameter, so the details of your question would be answered by the convergence of parameters of f' during training. $\endgroup$ – FauChristian Aug 31 '18 at 15:53
  • $\begingroup$ @Ryan, is the answer clear now? I'd like to make it so if not. $\endgroup$ – FauChristian Sep 4 '18 at 4:48
  • $\begingroup$ I think your answer is clear. I'm just still not quite seeing what the purpose of the normalization function, h, is. Why would I want to normalize the sum of the features? I know you deal with that for the individual functions that you present, but is the strategy to normalize to keep the sum that you feed in a reasonable range? Is there some other conceptual reason? BTW, I'm accepting your answer because it seems like the best one so far. $\endgroup$ – Ryan Sep 6 '18 at 14:28
  • $\begingroup$ @Ryan, h is not to alter ranges. Initialization of the first vector-matrix multiplication (learning parameters being the matrix) in the sequence of layers does that. Think of h as another kind of initialization that controls curvature. It too increases the likelihood and speed in which convergence occurs. $\endgroup$ – FauChristian Sep 12 '18 at 15:53
0
$\begingroup$

I can think of 4 options:

  1. One option is to divide the data so that each data point only has one picture but we have multiple data points per one real estate with structured data duplicated. Then calculate the forecasts and average the resulting predicted price between data points belonging to one real estate. Here we probably assume that the quality of pictures is more important than what is being shot.

  2. The other option is to divide the pictures to categories: bedroom1, bathroom, bedroom2, kitchen and so on. Then for the missing pictures use a black square. This way you will be able to have multiple pictures in one data point.

  3. The third option would be to have places for the maximum amount of pictures and fill only the first pictures that are available. And the other pictures fill with black squares/Nans.

  4. The best option would probably be combining option number two and three. So you have categories and maximum number of pictures per category. So you will have four categories for bathroom1 if the maximum is 4 pictures of the first bathroom:

    Bathroom1_1, Bathroom1_2, Bathroom1_3, Bathroom1_4, Bedroom1_1, Bedroom1_2, Bedroom1_3, Bedroom2_1, Bedroom2_2 ... and so on.

The gradient boosting algorithms are very good at working with missing values. So probably lightgbm and xgboost as last layers could give good results.

So in the end you can check all the variants and even do ensembles from the variants that give the best results.

$\endgroup$
  • $\begingroup$ option 2,3, and 4 is overkilling, you have useless datas (blank/black images) ! $\endgroup$ – Jérémy Blain Aug 23 '18 at 7:47
  • 1
    $\begingroup$ According to my experience, boosting algorithms, like lightgbm and xgboost, significantly increase performance if you use rows with missing values instead of just removing them. Missing value is also an information. If there are not all photos of all rooms then I would suggest that the seller is hiding something. And the price should go down because the missing room could be in awful conditions. $\endgroup$ – keiv.fly Aug 23 '18 at 7:57
  • $\begingroup$ Maybe, but I'm not sure I like the blank images either in option 4, for example, say that the kitchen is gorgeous, but your algorithm punishes a single picture of the kitchen because of the blank 5 slots for kitchen pictures. $\endgroup$ – Ryan Aug 23 '18 at 15:02
  • $\begingroup$ I am also concerned about option 1. I like that it deals with a test set observation that can have more than the max number of images in the training set, which is a real concern, but I do worry that averaging would wash out a picture that could severely move price in one direction or the other. $\endgroup$ – Ryan Aug 23 '18 at 15:05
  • 1
    $\begingroup$ Blank does not mean punishment. It is for the algorithm to decide which empty picture to punish. Or it can learn to just use one picture that is present. If in your dataset one picture of a gorgeous kitchen has a high price then the algorithm will learn that one picture is enough. $\endgroup$ – keiv.fly Aug 23 '18 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.