I have a regression task that I tray to solve with AI. I have around 6M rows with about 30 columns. (originally there was 100, but I reduce it with drop feature importance)
I understand basic principle: Look if model overfit or underfit - according change the parameters. In theory. I would ask for help with two graphs:
- If I understand correctly what is going on
- How would you attack the situation.
1. Graph
- I use LightGBM
- learning_rate = 1
- max_depth = 3
- num_leaves = 2**15,
- number of iterations = 4000
If I understand this model is Underfitting. The validation and training is falling, but not very much... BUT: The number of iteration is now too large and place higher number is not ok. Learning rate is 1 (as hight as it gets). Only the max_depth is low, but if it is higher (I try 30) the graph is same, just the values are worse.
So, what to do, so that model would not underfit.
- Graph
- I use Neural Nets
- epochs=200,
- batch_size=64
The model
i = Input(shape=(100,), name='input')
x = Dense(128)(i)
x = Dense(64)(i)
o = Dense(1, activation='relu', name='output')(x)
Here I am not sure. This doesn't really looks like underfit, but more that doesn't converged. Is this right?
So, should I create more complex model (more neurones or more layers?)?
And how much epochs do I need to see this behaviour? Because, in the beginning I use only 10 epochs, for faster development, and I thought that model is overfitting. Only when I use more epochs I see that I was wrong.
How would you start to "debug" this neural net? What would be the plan of attack?