Extracting information from RNA sequence

Question

I am relatively new to machine learning, and I am trying to use a deep neural network to extract some information from sequences of RNA.

A quick overview of RNA: there is both sequence and structure. I am currently expressing the sequence with one-hot encoding (so a sequence of length $60$ would be expressed as a $60 \times 4$ matrix, with one row for each letter of the sequence, and one column for each possible value of that letter). I am also feeding the 2D structure of the RNA into the network, which is expressed as $60 \times 60$ matrix for a sequence of length $60$.

I am trying to use these inputs to predict a single continuous value for a given sequence.

Currently, I am using pretty much the exact setup from this tutorial. I chose this architecture because it allows me to separate the inputs (sequence and structure) and have individual layers for them before merging them into a single model. I think this makes more sense than trying to glue the two separate pieces of data together into a single input.

However, the model doesn't seem to be learning anything - validation loss decreases very slightly then plateaus.

If anyone has suggestions, especially someone who has worked with RNA, DNA, or proteins before, I would really, really appreciate it. I am new to this, and I am not sure how to improve my model from here.

def create_mlp(height,width,filters=(16, 16, 32, 32, 64), regress=False):
    # initialize the input shape and channel dimension, assuming
  # TensorFlow/channels-last ordering
  inputShape = (height, width)
  chanDim = -1
  # define the model input
  inputs = Input(shape=inputShape)
  # loop over the number of filters
  for (i, f) in enumerate(filters):
    # if this is the first CONV layer then set the input
    # appropriately
    if i == 0:
      x = inputs
    # CONV => RELU => BN => POOL
    x = Conv1D(f, 3, padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling1D(pool_size=2)(x)
  # flatten the volume, then FC => RELU => BN => DROPOUT
  print(x.shape)
  x = Flatten()(x)
  x = Dense(16)(x)
  x = Activation("relu")(x)
  x = BatchNormalization(axis=chanDim)(x)
  x = Dropout(0.5)(x)
  # apply another FC layer, this one to match the number of nodes
  # coming out of the MLP
  x = Dense(4)(x)
  x = Activation("relu")(x)
  # check to see if the regression node should be added
  if regress:
    x = Dense(1, activation="linear")(x)
  # construct the CNN
  model = Model(inputs, x)
  # return the CNN
  return model

def create_cnn(width, height, depth, filters=(16, 16, 32, 32, 64), regress=False):
  # initialize the input shape and channel dimension, assuming
  # TensorFlow/channels-last ordering
  inputShape = (height, width, depth)
  chanDim = -1
  # define the model input
  inputs = Input(shape=inputShape)
  # loop over the number of filters
  for (i, f) in enumerate(filters):
    # if this is the first CONV layer then set the input
    # appropriately
    if i == 0:
      x = inputs
    # CONV => RELU => BN => POOL
    x = Conv2D(f, (3, 3), padding="same")(x)
    x = Activation("relu")(x)
    x = BatchNormalization(axis=chanDim)(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
  # flatten the volume, then FC => RELU => BN => DROPOUT
  print(x.shape)
  x = Flatten()(x)
  x = Dense(16)(x)
  x = Activation("relu")(x)
  x = BatchNormalization(axis=chanDim)(x)
  x = Dropout(0.5)(x)
  # apply another FC layer, this one to match the number of nodes
  # coming out of the MLP
  x = Dense(4)(x)
  x = Activation("relu")(x)
  # check to see if the regression node should be added
  if regress:
    x = Dense(1, activation="linear")(x)
  # construct the CNN
  model = Model(inputs, x)
  # return the CNN
  return model

mlp = create_mlp(l, 4, regress=False)
  cnn = create_cnn(l, l, 1, regress=False)
  # create the input to our final set of layers as the *output* of both
  # the MLP and CNN
  #cnn.output.reshape()
  combinedInput = concatenate([mlp.output, cnn.output])

Answer 1

The link you have mentioned is using Dense layers. One thing to start with would be to use 1D CNNs (they will capture some local information). Also, since sequence matters in your case, refrain from one-hot encodings (just 1, 2, 3, 4). And for the 2D matrix, use a 2D CNN. Then, flatten your encoding for both 1D CNN and 2D CNN, then, finally combine them. This shall give you some improvement.

This is for your network your are already using.

Here is an alternate suggestion for you:

Look at GCN (Graph Convolutional Network). Since, you have a graph structure (60x60 matrix is actually an adjacency matrix) and each node i.e. 60x4 in your case, can then become node features. This is what you want a GNN and its variants for. Look at GAT (Graph Attention Networks) if GCN don't work for you.