Is there ever a need to combine deep learning frameworks? (Eg. TensorFlow & Torch)?

Question

Imagine a simple scenario of having a large repository using one framework and integrated with data/robots, etc., then having a new feature requested and the framework missing some vital functionality that is available in another framework (say a new kind of layer).

For many mathematical libraries, it can be easy to reverse engineer the specific function that one would want to form the other framework so as to not import the entire framework but in the case of deep learning, this isn't so trivial nor that easily testable.

In these cases, does one take on the problem of reverse engineering the functionality or do they attempt to combine the architectures? If it's the combination, what are the biggest issues with integration?

Answer 1

In the time since I asked this question, I have been able to combine Tensorflow and Chainer considerably well. That being said, one should try to avoid combining deep learning frameworks if one can for a few reasons:

1. It doubles the amount of documentation one needs to reference
2. It makes it difficult for new developers to become familiar with the code base.
3. It creates multiple data transfer bottlenecks when one has to transfer data between the deep learning different configurations such as: CPU -> GPU (Tensorflow) -> CPU -> GPU (Chainer) -> CPU ... etc.

That being said, I combined them because of time constraints on a research project which would not be finding its way into an industrial product. In this research setting, I still wouldn't recommend combining frameworks if one can avoid it but if it means doubling the time it takes to do the project for a proof of concept or experiment, then it seems reasonable and can drastically speed up development until the missing functionality is added to the desired framework.

There are 2 issues with integration:

• Abstracting the networks so that the main training/testing script does not depend on the framework.

  This is important in the event that the missing functionality of your desired framework is added. It also helps new developers understand what the overall algorithm or experiment is without being bogged down by framework details.

• Tensorflow and similar frameworks which require a Session object.

  Chainer, PyTorch (which came from Chainer), and any other framework that doesn't require a something like Tensorflow's Session all play very nicely together and can be used (almost) interchangeably. For Session based frameworks, the main script/loop/experiment must all take place within a Session. With proper abstraction, my experiment only required a with tf.Session() as sess: to be used in the main experiment and was the only reference that Tensorflow was being used.

Although I still don't recommend combining frameworks for the reasons specified above, if one abstracts their framework calls enough, it is almost seamless to integrate deep learning frameworks.