0
$\begingroup$

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow.

I am trying to train a model on sequential data [shape:((435802, 20), (435802,))] where every sequence is of length 20 and $$X,Y \in [0,6]$$

Concept: each value in the sequence represents a class of real data instance and the objective is to predict the next class instance.

X[n] = [3, 4, 3, 0, 2, 6, 6, 6, 6, 5, 4, 3, 2, 2, 2, 2, 4, 2, 0, 0] => Y[n] = 0

Class Distribution in Y:

0: 46458
1: 40909
2: 76398
3: 102515
4: 80830
5: 43569
6: 45123

I have tried various combinations of layers like Conv, LSTM, SimpleRNN, Bidirectional, GRU, Dense, etc

Latest Model Definition:

model: "sequential_69"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv1d_11 (Conv1D)          (None, 20, 64)            256       
                                                                 
 bidirectional_32 (Bidirecti  (None, 20, 512)          494592    
 onal)                                                           
                                                                 
 bidirectional_33 (Bidirecti  (None, 20, 512)          1182720   
 onal)                                                           
                                                                 
 lstm_56 (LSTM)              (None, 20, 64)            147712    
                                                                 
 lstm_57 (LSTM)              (None, 20, 32)            12416     
                                                                 
 lstm_58 (LSTM)              (None, 16)                3136      
                                                                 
 dropout_95 (Dropout)        (None, 16)                0         
                                                                 
 flatten_42 (Flatten)        (None, 16)                0         
                                                                 
 dropout_96 (Dropout)        (None, 16)                0         
                                                                 
 dense_217 (Dense)           (None, 7)                 119       
                                                                 
=================================================================
Total params: 1,840,951
Trainable params: 1,840,951
Non-trainable params: 0
_________________________________________________________________
Epoch 1/20
698/698 - 24s - loss: 1.8671 - accuracy: 0.2324 - val_loss: 1.9549 - val_accuracy: 0.1769 - 24s/epoch - 35ms/step
Epoch 2/20
698/698 - 12s - loss: 1.8484 - accuracy: 0.2451 - val_loss: 1.9464 - val_accuracy: 0.1835 - 12s/epoch - 18ms/step
Epoch 3/20
698/698 - 12s - loss: 1.8189 - accuracy: 0.2539 - val_loss: 1.8762 - val_accuracy: 0.2491 - 12s/epoch - 17ms/step
Epoch 4/20
698/698 - 12s - loss: 1.7844 - accuracy: 0.2726 - val_loss: 1.8567 - val_accuracy: 0.2470 - 12s/epoch - 17ms/step
Epoch 5/20
698/698 - 12s - loss: 1.7129 - accuracy: 0.3054 - val_loss: 1.7388 - val_accuracy: 0.3015 - 12s/epoch - 18ms/step
Epoch 6/20
698/698 - 12s - loss: 1.5867 - accuracy: 0.3488 - val_loss: 1.5342 - val_accuracy: 0.3941 - 12s/epoch - 17ms/step
Epoch 7/20
698/698 - 12s - loss: 1.4544 - accuracy: 0.3974 - val_loss: 1.5511 - val_accuracy: 0.3502 - 12s/epoch - 17ms/step
Epoch 8/20
698/698 - 12s - loss: 1.3705 - accuracy: 0.4300 - val_loss: 1.4285 - val_accuracy: 0.4169 - 12s/epoch - 17ms/step
Epoch 9/20
698/698 - 12s - loss: 1.3284 - accuracy: 0.4489 - val_loss: 1.3488 - val_accuracy: 0.4606 - 12s/epoch - 17ms/step
Epoch 10/20
698/698 - 12s - loss: 1.3064 - accuracy: 0.4587 - val_loss: 1.3678 - val_accuracy: 0.4483 - 12s/epoch - 17ms/step
Epoch 11/20
698/698 - 12s - loss: 1.2945 - accuracy: 0.4640 - val_loss: 1.3417 - val_accuracy: 0.4631 - 12s/epoch - 17ms/step
Epoch 12/20
698/698 - 12s - loss: 1.2848 - accuracy: 0.4694 - val_loss: 1.3644 - val_accuracy: 0.4471 - 12s/epoch - 18ms/step
Epoch 13/20
698/698 - 12s - loss: 1.2780 - accuracy: 0.4729 - val_loss: 1.3474 - val_accuracy: 0.4616 - 12s/epoch - 17ms/step
Epoch 14/20
698/698 - 12s - loss: 1.2717 - accuracy: 0.4741 - val_loss: 1.3396 - val_accuracy: 0.4604 - 12s/epoch - 18ms/step
Epoch 15/20
698/698 - 12s - loss: 1.2666 - accuracy: 0.4764 - val_loss: 1.3862 - val_accuracy: 0.4390 - 12s/epoch - 18ms/step
Epoch 16/20
698/698 - 12s - loss: 1.2630 - accuracy: 0.4773 - val_loss: 1.3143 - val_accuracy: 0.4672 - 12s/epoch - 17ms/step
Epoch 17/20
698/698 - 12s - loss: 1.2582 - accuracy: 0.4788 - val_loss: 1.3189 - val_accuracy: 0.4640 - 12s/epoch - 17ms/step
Epoch 18/20
698/698 - 12s - loss: 1.2571 - accuracy: 0.4796 - val_loss: 1.3443 - val_accuracy: 0.4549 - 12s/epoch - 17ms/step
Epoch 19/20
698/698 - 12s - loss: 1.2529 - accuracy: 0.4816 - val_loss: 1.3186 - val_accuracy: 0.4692 - 12s/epoch - 17ms/step
Epoch 20/20
698/698 - 12s - loss: 1.2504 - accuracy: 0.4819 - val_loss: 1.3074 - val_accuracy: 0.4714 - 12s/epoch - 17ms/step
313/313 [==============================] - 4s 7ms/step
              precision    recall  f1-score   support

           0       0.62      0.80      0.70      1059
           1       0.46      0.14      0.22       986
           2       0.43      0.46      0.44      1760
           3       0.46      0.56      0.50      2310
           4       0.44      0.48      0.46      1890
           5       0.41      0.09      0.15       941
           6       0.59      0.76      0.66      1054

    accuracy                           0.49     10000
   macro avg       0.49      0.47      0.45     10000
weighted avg       0.48      0.49      0.46     10000

Accuracy Graph

Some more context: I am a noob in terms of Neural Networks training and have tried basic models like TF examples, Pytorch Examples and some AutoEncoder examples.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Can you elaborate on the relationship between X[n] and X[n-1] ?

is there a f so that X[n] = f(X[n-1]) ? is each X[n] the output of different sources or one single source ? is it the evolution of a state machine ?

Have you analyzed your database ? It is not balanced, but do you have things like a same sequence associated to different targets ?

Improve this answer
$\endgroup$
2
  • $\begingroup$ before training, i do implement rebalancing by resampling. as far as the multiple next sequences to same sequence...there are a few samples like that. but only a few $\endgroup$
    – Th3Nic3Guy
    Commented Sep 16, 2023 at 8:00
  • $\begingroup$ Also I have not done the F(x) analysis. will check on the state machine $\endgroup$
    – Th3Nic3Guy
    Commented Sep 16, 2023 at 8:02
0
$\begingroup$

from the details which you have share ,i can see following things can be reasons for less accuracy

1. Class imbalance

in the class distribution the classes 0, 3, and 4 have many instances as compared to classes 1 and 5, solving this imbalance can improve the accuracy of the model

You can do it by following ways

=> Weighted Loss Function:

You can provide class weights when compiling the model to give more importance to underrepresented classes:

for example, consider this code block

from sklearn.utils import class_weight
class_weights = class_weight.compute_class_weight('balanced', classes=np.unique(Y), y=Y)
model.fit(X_train, y_train, class_weight=dict(enumerate(class_weights)))

=> You can use methods like resampling to oversample the minority classes or undersample the majority classes.

=> You can apply the data augmentation by adding noise, applying time warping, or even using techniques like jittering

2. Regularization

=> Adding Regularization can be a good way to reduce overfitting you can achieve it by adding dropout layers

for example

from tensorflow.keras.regularizers import l2
model.add(Dense(64, kernel_regularizer=l2(0.001)))

3. Tuning Model Architecture

Your current model architecture is complex, and simplifying or adjusting certain layers could help can achieve it by following steps:

  • Reduce Complexity of Bidirectional Layers: Since you're using multiple bidirectional layers, reducing the number of units in each LSTM layer might help with generalization.

  • Skip the Flatten Layer: Removing the Flatten layer and feeding the final LSTM output directly into a Dense layer can improve performance in sequential models.

  • Attention Mechanism: Consider adding an attention layer to help the model focus on specific parts of the input sequence, which can improve predictive accuracy in sequential tasks.

4. Learning Rate Tuning or changing optimizer

=> Learning Rate Tuning: Try using a learning rate scheduler or manually tune the learning rate. A learning rate that is too high might prevent the model from converging.

lr_schedule = tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-3 * 10**(epoch / 20))

=> Use AdamW Optimizer: The AdamW optimizer includes weight decay, which can help with generalization.

5. Hyperparameter Tuning

=> You can experiment with hyperparameters like batch size, number of LSTM units, number of layers, and activation functions. Use grid search or random search with KerasTuner to find optimal hyperparameters. KerasTuner is a get tool for experimenting with hyperparameters

and for the evaluation part, I suggest you use the Confusion Matrix Analyze where the model is making errors by inspecting the confusion matrix. This can highlight which classes are causing problems, helping you focus on improving those predictions.

For more inforation you can check these links

=> https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler

=> https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout

=> https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/AdamW

=> https://keras.io/keras_tuner/

=> https://maddevs.io/writeups/basic-data-augmentation-method-applied-to-time-series/

=> https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .