Can neural networks always be assembled like Lego blocks?

BACKGROUND

Consider a supervised problem which is based on

• two scalar features (1) and (2) as well as
• a third, "time-dependent", feature consisting of a sequence of five values (3)-(7).

For example, (1) and (2) could be the revenue and rating of a stock, respectively, while (3) to (7) are its closing prices for the past five days. The target is whether the price will go up or down on the following day.

Let's assume we want to use a deep neural network (DNN) architecture where the sequential features are processed by a recurrent neural network (RNN). Schematically, a holistic approach could be as follows

                    *     [target]
o o o   [3rd DNN layer for merging]
[RNN embedding] ¤ ¤ ¤ o o [2nd DNN layer for ① & ②]
\|/  o o [1st DNN layer for ① & ②]
[RNN] ③→④→⑤→⑥→⑦   ① ② [scalar features]


where the scalar features are processed by two 2-node DNN layers and the sequential features are embedded by, say, an LSTM cell into a single 3-node layer (denoted above as ¤ ¤ ¤). A final, 3-node layer merges the embeddings before, say, soft-maxing into the target.

The configuration above is holistic in that back-propagation works all the way form the target back to the "raw" features (1)-(7) in one go. There is no pre-training or transfer learning to speak of.

QUESTION

Instead of the above, is there any reason why one should (or shouldn't) break down the neural network into smaller DNN blocks, just like Lego bricks? Namely, one could extract the RNN as a standalone DNN

          [target] *
[RNN embedding] ø ø ø
\|/
[RNN] ③→④→⑤→⑥→⑦


and train it directly on the target. That trained RNN could be used to produce the embedding ø ø ø which in turn would a posteriori be merged with the output for the two-layered DNN that processes the scalar features.

More concisely, will we always have ø ø ø == ¤ ¤ ¤? If not, what should one keep in mind when pondering holistic vs. modular architectures of neural networks?

Any other thoughts?