"Porpoising" in latter stages of validation loss and MSE charts in Keras

Question

Performing a prediction of a continuous y target using Keras, the simple structure of the code revolves around;

model = Sequential()  
model.add(Dense(200, input_dim=15, activation= "relu"))  
model.add(Dense(750, activation= "relu"))  
model.add(Dense(500, activation= "relu"))  
model.add(Dense(750, activation= "relu"))  
model.add(Dense(500, activation= "relu"))  
model.add(Dense(200, activation= "relu"))  
model.add(Dense(100, activation= "relu"))  
model.add(Dense(50, activation= "relu"))  
model.add(Dense(1)) 

model.compile(loss= 'mse' , optimizer='adam', metrics=['mse','mae'])  
history=model.fit(X_train, y_train, batch_size=50,  epochs=150,  
                  verbose=1, validation_split=0.2)

This has resulted in the following metric chart;

What might be causing these, and how to eliminate (or greatly reduce) them?

UPDATE: Just reduced the learning rate to 0.0001 per Neil Slater's suggestion, and the loss curve may have had the spikes reduced, though the scale of the graph has changed. The training loss has increased from 0.00007 to 0.00037, the validation loss from 0.0014 to 0.002, and the prediction error increased from 0.037 to 0.046.

I then changed epsilon from it's value of 1e-07 to 0.1 and increased the epochs from 150 to 500. The validation loss increased to 0.0082 and the prediction error increased to 0.093, with the corresponding model loss shown below.

While not an overall improvement at either step, this did remove the spikes as I requested, hence Neil's advice gives me additional considerations to explore and measure within the Adam optimizer (along with other optimizers), so I consider this to have been an important learning experience. One such exploration uncovered this more detailed explanation of optimizers than I had been exposed to before, as well as this 3D visualization of loss topologies and the effect differing optimizers and parameters have on finding the optimal minima (keep a sharp eye on the options being chosen in the upper right corner).

Answer 1

What might be causing these, and how to eliminate (or greatly reduce) them?

It is difficult to be sure just from the graph, but I note you are using the Adam optimiser. It shares with a few other optimisers (most notably RMSProp) that it divides current gradients by a rolling mean of recent gradients to set step sizes. This can cause some minor instability when gradients get close to zero for a long while before growing again. This might happen at a saddle point where only some fraction of the network parameters are critical to results for a few iterations before hitting some other direction of slope where changing the "quiet" parameters becomes important again. This is more likely to occur as loss values become small, and close to perfect convergence.

There are a couple of hyperparameters in Adam that may reduce the effect:

• Reduce learning rate. The default learning rate in Adam is often set to 0.001, but you will find a lot of researchers will reduce that to magnitudes around 0.0001 or 0.00001

• Increase epsilon. This is effectively the minimum rolling average gradient. The default is 1e-7 but it sometimes needs to be increased significantly, it depends on the loss surface. The official documentation suggests that it may even be useful to increase it up to 1.0