# Bounding Box Regression - An Adventure in Failure

I've solved many problems with neural networks, but rarely work with images. I have about 18 hours into creating a bounding box regression network and it continues to utterly fail. With some loss functions it will claim 80% accuracy during training and validation (with a truly massive loss on both) but testing the predictions reveals a bounding box that only moves one or two pixels in any given direction and seems to totally ignore the data. I've now implemented a form of IoU loss, but find that IoU is pinned at zero... which is obviously true based on the outputs after training. :). I'd like someone to look this over and give me some advice on how to proceed next.

## What I Have

I am generating 40000 examples of 200x100x3 images with a single letter randomly placed in each. Simultaneously I am generating the ground truth bounding boxes for each training sample. I have thoroughly validated that this all works and the data is correct.

## What I Do To It

I am then transforming the 200x100x3 images down to greyscale to produce a 200x100x1 image. The images are then normalized and the bounding boxes are scaled to fall between 0 and 1. In simplified form, this happens:

x_train_normalized = (x_data - 127.5) / 127.5
y_train_scaled = boxes[:TRAIN]/[WIDTH,HEIGHT,WIDTH,HEIGHT]


I've been through this data carefully, even reconstituting images and bounding boxes from it. This is definitely working.

## Training

To train, after trying mse and many others, all of which fail equally badly, I have implemented a simple custom IOU loss function. It actually returns -ln(IoU). I made this change based on a paper since the loss was (oddly?) pinned at zero over multiple epochs.

(Loss function:)

import tensorflow.keras.backend as kb
def iou_loss(y_actual,y_pred):
b1 = y_actual
b2 = y_pred
#    tf.print(b1)
#    tf.print(b2)
zero = tf.convert_to_tensor(0.0, b1.dtype)
b1_ymin, b1_xmin, b1_ymax, b1_xmax = tf.unstack(b1, 4, axis=-1)
b2_ymin, b2_xmin, b2_ymax, b2_xmax = tf.unstack(b2, 4, axis=-1)
b1_width = tf.maximum(zero, b1_xmax - b1_xmin)
b1_height = tf.maximum(zero, b1_ymax - b1_ymin)
b2_width = tf.maximum(zero, b2_xmax - b2_xmin)
b2_height = tf.maximum(zero, b2_ymax - b2_ymin)
b1_area = b1_width * b1_height
b2_area = b2_width * b2_height

intersect_ymin = tf.maximum(b1_ymin, b2_ymin)
intersect_xmin = tf.maximum(b1_xmin, b2_xmin)
intersect_ymax = tf.minimum(b1_ymax, b2_ymax)
intersect_xmax = tf.minimum(b1_xmax, b2_xmax)
intersect_width = tf.maximum(zero, intersect_xmax - intersect_xmin)
intersect_height = tf.maximum(zero, intersect_ymax - intersect_ymin)
intersect_area = intersect_width * intersect_height

union_area = b1_area + b2_area - intersect_area
iou = -1 * tf.math.log(tf.math.divide_no_nan(intersect_area, union_area))
return iou


### The Network

This has been through many, many iterations. As I said, I've solved many other problems with NNs... This is the first one to get me completely stuck. At this point, the network is dramatically stripped down but continues to fail to train at all:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, optimizers

tf.keras.backend.set_floatx('float32') # Use Float32s for everything

input_shape = x_train_normalized.shape[-3:]
model = keras.Sequential()
model.add(layers.Conv2D(4, 16, activation = tf.keras.layers.LeakyReLU(alpha=0.2), input_shape=input_shape))
model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=(2, 2)))
model.add(layers.Dropout(0.2))
model.add(layers.Flatten())
model.add(layers.Dense(200, activation = tf.keras.layers.LeakyReLU(alpha=0.2)))
model.add(layers.Dense(64, activation=tf.keras.layers.LeakyReLU(alpha=0.2)))
model.add(layers.Dense(4, activation="sigmoid"))

model.compile(loss = iou_loss, optimizer = "adadelta", metrics=['accuracy'])
history = model.fit(x_train_normalized, y_train_scaled, epochs=8, batch_size=100, validation_split=0.4)


All pointers are welcome! In the meantime I'm implementing a center point loss function to see if that helps at all.

• Still looking for suggestions and observations. I wrote a exponential differential distance loss function that examines the center points and it appears to be training better... We'll see when it finishes. – David Hoelzer Jan 19 at 16:39

## 1 Answer

In the end, this problem turned out to be largely a matter of the gradient descent falling into local minima.

For those reading for posterity, one of the issues in ML that is difficult to work around is that we cannot intuitively choose reasonable initial values for the weights, biases, and kernels (in the CNN). As a result, we typically allow them to initialize randomly. This can present some challenges.

One of the biggest challenges is that when you start from a random starting point, it's difficult to tell someone how to completely replicate your experiments. This isn't terribly important in the end since you can provide them with the saved parameters from your trained model. However, this can also lead to networks that appear to be "bad" that are in fact perfectly fine.

In this case, I had spent much of the time initializing the CNN with a uniform initializer (not present in the code above). I will sometimes use a random seed or some other function to generate initial values so that I can better improve networks through genetic search tools.

It seems that the uniform initializers combined with the various network iterations and this particular data lead to absolutely abysmal training performance and non-convergence.

When I ran the network as above with random initializations and one or two tweaks, it converged well. Some training iterations will pin one of the sides of the bounding box at the edge, some will never converge, but I've managed to successfully train several that are in the 96-98% accuracy range for the bounding boxes in my test set of 20000, so all is well!

• When you say "When I ran the network as above with random initializations and one or two tweaks", which specific initialization technique did you use in the end? Note that uniform initialization is also random. What tweaks did you use? – nbro Jan 21 at 1:56
• I had been using hm_uniform. I simply allowed the default initialization to take place. I need to plug this into one of my genetic search algorithms after finding a good seed that produces good results to better refine the model. – David Hoelzer Jan 21 at 12:51