0
$\begingroup$

I am relatively new to machine learning, and I am trying to use a deep neural network to extract some information from sequences of RNA.

A quick overview of RNA: there is both sequence and structure. I am currently expressing the sequence with one-hot encoding (so a sequence of length $60$ would be expressed as a $60 \times 4$ matrix, with one row for each letter of the sequence, and one column for each possible value of that letter). I am also feeding the 2D structure of the RNA into the network, which is expressed as $60 \times 60$ matrix for a sequence of length $60$.

I am trying to use these inputs to predict a single continuous value for a given sequence.

Currently, I am using pretty much the exact setup from this tutorial. I chose this architecture because it allows me to separate the inputs (sequence and structure) and have individual layers for them before merging them into a single model. I think this makes more sense than trying to glue the two separate pieces of data together into a single input.

However, the model doesn't seem to be learning anything - validation loss decreases very slightly then plateaus.

If anyone has suggestions, especially someone who has worked with RNA, DNA, or proteins before, I would really, really appreciate it. I am new to this, and I am not sure how to improve my model from here.

def create_mlp(height,width,filters=(16, 16, 32, 32, 64), regress=False):
    # initialize the input shape and channel dimension, assuming
  # TensorFlow/channels-last ordering
  inputShape = (height, width)
  chanDim = -1
  # define the model input
  inputs = Input(shape=inputShape)
  # loop over the number of filters
  for (i, f) in enumerate(filters):
    # if this is the first CONV layer then set the input
    # appropriately
    if i == 0:
      x = inputs
    # CONV => RELU => BN => POOL
    x = Conv1D(f, 3, padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling1D(pool_size=2)(x)
  # flatten the volume, then FC => RELU => BN => DROPOUT
  print(x.shape)
  x = Flatten()(x)
  x = Dense(16)(x)
  x = Activation("relu")(x)
  x = BatchNormalization(axis=chanDim)(x)
  x = Dropout(0.5)(x)
  # apply another FC layer, this one to match the number of nodes
  # coming out of the MLP
  x = Dense(4)(x)
  x = Activation("relu")(x)
  # check to see if the regression node should be added
  if regress:
    x = Dense(1, activation="linear")(x)
  # construct the CNN
  model = Model(inputs, x)
  # return the CNN
  return model

def create_cnn(width, height, depth, filters=(16, 16, 32, 32, 64), regress=False):
  # initialize the input shape and channel dimension, assuming
  # TensorFlow/channels-last ordering
  inputShape = (height, width, depth)
  chanDim = -1
  # define the model input
  inputs = Input(shape=inputShape)
  # loop over the number of filters
  for (i, f) in enumerate(filters):
    # if this is the first CONV layer then set the input
    # appropriately
    if i == 0:
      x = inputs
    # CONV => RELU => BN => POOL
    x = Conv2D(f, (3, 3), padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
  # flatten the volume, then FC => RELU => BN => DROPOUT
  print(x.shape)
  x = Flatten()(x)
  x = Dense(16)(x)
  x = Activation("relu")(x)
  x = BatchNormalization(axis=chanDim)(x)
  x = Dropout(0.5)(x)
  # apply another FC layer, this one to match the number of nodes
  # coming out of the MLP
  x = Dense(4)(x)
  x = Activation("relu")(x)
  # check to see if the regression node should be added
  if regress:
    x = Dense(1, activation="linear")(x)
  # construct the CNN
  model = Model(inputs, x)
  # return the CNN
  return model

mlp = create_mlp(l, 4, regress=False)
  cnn = create_cnn(l, l, 1, regress=False)
  # create the input to our final set of layers as the *output* of both
  # the MLP and CNN
  #cnn.output.reshape()
  combinedInput = concatenate([mlp.output, cnn.output])
$\endgroup$
3
  • $\begingroup$ Hello. Welcome to Artificial Intelligence Stack Exchange. Can you please clarify what you're trying to predict for each sequence/RNA? Moreover, can you clarify why your network is a $60 \times 60$ matrix? This means that if you multiply your input matrix $60 \times 4$ from the right with your first weight matrix you will get another $60 \times 4$ matrix. If this is correct, what do you do then with this new $60 \times 4$ matrix? To get a final number (which is what you're doing, given that you're doing regression), you need to parse this matrix, but how? $\endgroup$ – nbro Feb 10 at 10:21
  • $\begingroup$ Additionally, I would also suggest that you show us the plot of the validation loss, specify how many epochs you trained your model for, what is the size of your training and validation dataset, learning rate, number of parameters of the net, and any other metric you're computing to assess the quality of the predictions/regressions. Finally, please, try to ask a specific question and not just "any suggestions to solve this problem are welcome", then put it in the title so that people immediately understand what you're asking here. $\endgroup$ – nbro Feb 10 at 10:23
  • $\begingroup$ Sorry if these are too many questions at the same time, but sometimes, even though we think that our post/question is clear to us, it's not enough to clear to other people (at least all the possible details that may be helpful to provide an answer may not be available). $\endgroup$ – nbro Feb 10 at 10:26
1
$\begingroup$

The link you have mentioned is using Dense layers. One thing to start with would be to use 1D CNNs (they will capture some local information). Also, since sequence matters in your case, refrain from one-hot encodings (just 1, 2, 3, 4). And for the 2D matrix, use a 2D CNN. Then, flatten your encoding for both 1D CNN and 2D CNN, then, finally combine them. This shall give you some improvement.

This is for your network your are already using.

Here is an alternate suggestion for you:

Look at GCN (Graph Convolutional Network). Since, you have a graph structure (60x60 matrix is actually an adjacency matrix) and each node i.e. 60x4 in your case, can then become node features. This is what you want a GNN and its variants for. Look at GAT (Graph Attention Networks) if GCN don't work for you.

$\endgroup$
2
  • $\begingroup$ Dear Abhishek, thank you for your response! I actually have tried 1D CNN for the sequence and 2D CNN. I want to add a convolutional layer after I combine them, but I am having trouble. Do you know how to do this? I have added the relevant code to my original question $\endgroup$ – gollyzoom Feb 16 at 7:56
  • $\begingroup$ Put combined_input through a Dense layer. Also, return Flattened tensors (See Flatten Layer) from both the functions. $\endgroup$ – Abhishek Verma Feb 16 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.