0
$\begingroup$

Here is a model that trains time series data in (batch, step, features) way.

I have kept the random state for train test split function the same. Every parameter below the same, running the model training yields different outcomes every time and the outcomes are drastically different.

What may be the factors that led to this? Regularization?

outcome_1 outcome_2 outcome_3

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=666)

def attention_model(X_train, y_train, X_test, y_test,num_classes,dropout=0.2, batch_size=68, learning_rate=0.0001,epochs=20,optimizer='Adam'):
    
    Dense_unit = 12
    LSTM_unit = 12
    
    attention_param = LSTM_unit*2
    attention_init_value = 1.0/attention_param
    
    
    u_train = np.full((X_train.shape[0], attention_param),
                      attention_init_value, dtype=np.float32)
    u_test = np.full((X_test.shape[0],attention_param),
                     attention_init_value, dtype=np.float32)
    
    
    with keras.backend.name_scope('BLSTMLayer'):
        # Bi-directional Long Short-Term Memory for learning the temporal aggregation
        input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
        x = Masking(mask_value=0)(input_feature)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)


        y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.000029),kernel_regularizer=l2(0.027),recurrent_regularizer=l2(0.025),return_sequences=True, dropout=dropout))(x)
#         y = Bidirectional(LSTM(LSTM_unit, kernel_regularizer=l2(0.01),recurrent_regularizer=l2(0.01), return_sequences=True, dropout=dropout))(y)

    with keras.backend.name_scope('AttentionLayer'):
        # Logistic regression for learning the attention parameters with a standalone feature as input
        input_attention = Input(shape=(LSTM_unit * 2,))
        u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)

        # To compute the final weights for the frames which sum to unity
        alpha = dot([u, y], axes=-1)  # inner prod.
        alpha = Activation('softmax')(alpha)

    with keras.backend.name_scope('WeightedPooling'):
        # Weighted pooling to get the utterance-level representation
        z = dot([alpha, y], axes=1)

    # Get posterior probability for each emotional class
    output = Dense(num_classes, activation='softmax')(z)

    model = Model(inputs=[input_attention, input_feature], outputs=output)

    optimizer = opt_select(optimizer,learning_rate)
    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)


    hist = model.fit([u_train, X_train], 
                     y_train, 
                     batch_size=batch_size, 
                     epochs=epochs, 
                     verbose=2, 
                     validation_data=([u_test, X_test], y_test))
    

#kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),
    return hist

batch_size= 150
#217
epochs = 1000
learning_rate = 0.00081
optimizer = 'RMS'
num_classes = y_train.shape[1]
dropout=0.22

tf.keras.backend.clear_session()

history = attention_model(X_train, y_train, X_test, y_test, num_classes,dropout = dropout,batch_size=batch_size, learning_rate=learning_rate,epochs=epochs,optimizer=optimizer
)
$\endgroup$
2
  • $\begingroup$ Hello. Welcome to AI SE. I don't know if this is "just" a programming issue or not, but I would like you to note that programming questions (like "Why am I getting this programming error/bug?") are generally off-topic here. These questions are better suited for Stack Overflow. I don't know if your question is a pure programming issue (because I didn't read it, but it contains code...), but just keep this in mind. You should read ai.stackexchange.com/help/on-topic for more info. $\endgroup$ – nbro Jan 26 at 15:05
  • 1
    $\begingroup$ @nbro Hi nbro, I was not sure what was causing the problem. My initial thought was that the problem was caused by some randomness of models not because of codes so I post it here. $\endgroup$ – Leo Jan 27 at 1:10
0
$\begingroup$

By default, Keras sets shuffle argument True, so you should set the numpy seed before importing Keras.

CPU

from numpy.random import seed
seed(25)
from keras.models import Sequential

GPU

tf.random.set_seed(seed)
$\endgroup$
3
  • $\begingroup$ Hi yahkyo, I set the random seed to a fixed value and rerun the whole model but the outcomes still changed every training. So I then set the shuffle = False, the outcomes were still not exactly the same but this time the trends were generally the same. $\endgroup$ – Leo Jan 26 at 5:50
  • $\begingroup$ Hi Leo, I think random shuffle in your code is on GPU. However numpy works on CPU. Have you tried tf.random.set_seed(seed) ? $\endgroup$ – yakhyo_ Jan 26 at 5:58
  • $\begingroup$ You are right, I was running GPU. tf.random.set_seed(seed) this code doesn't give the 100% same outcomes, but I think 95% similarity is good for parameter adjustment. Thanks. $\endgroup$ – Leo Jan 26 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.