4
$\begingroup$

I've been using PyTorch to do research for a while and it seems to be quite easy to implement new things with. Also, it is easy to learn and I didn't have any problem with following other researchers code so far.

However, I wonder whether TensorFlow has any advantage over PyTorch. The only advantage I know is, it's slightly faster than PyTorch.

In general, does TensorFlow have any concrete advantages over PyTorch apart from performance, in particular for research purposes?

$\endgroup$
  • 2
    $\begingroup$ The only advantage I can think of is that since many good ML models are produced by Google engineers and scientists on Google's payroll, and hence they implement models on TensorFlow. Thus, you can easily find a well known model on TensorFlow whereas you have to rely on the community to make it available in PyTorch. $\endgroup$ – DuttaA Feb 12 at 2:17
  • 1
    $\begingroup$ In my company we do it all in Keras and TF 1.x, and every single time I had to debug an operation embedded in the graph (say loss function or custom layers...) it was a real pain. I don't know if TF 2.x fixes that. The only thing I find interesting about TF approach is the cheap TPUs for training. $\endgroup$ – JVGD Feb 12 at 13:19
4
$\begingroup$

In the past, I have used TensorFlow (1 and 2), Keras and PyTorch, so I will give an answer based on my experience. Currently, I use TF 2 and Keras (the version shipped with TF 2).

In my (but not only) opinion, TF 1 is really ugly and painful, given that it involves sessions, placeholders and, in general, you need to define the computational graph before executing anything (even the simplest programs). With TF 2, you don't need sessions and placeholders anymore, and this is a very big improvement because you don't need to think that you first need to define a computational graph and then feed it with the input. Essentially, in TF 2, you can write TF code and you can almost be sure that your TF 2 code will look like any other Python code.

TF and Keras have some bugs, but I think that also PyTorch must have some bugs (even though I don't remember having encountered them).

TF comes with TensorBoard. You can also easily use TensorBoard from Keras. About one year ago, I had written a blog post about visualization tools for PyTorch. At the time, the tools didn't seem as developed as TensorBoard, but there were already a few options.

Debugging in TF 2 isn't still perfect, but I would say that it has improved with respect to TF 1 (mainly because there is less boilerplate code). To debug in TF 2.0, I mainly use tf.print (which behaves similarly to Python's print function), but there may be more advanced tools (e.g. I think there's a TF debugging plugin for PyCharm), even though I've never used them. TensorBoard could also be used as a debugging tool (for example, to understand how your models are learning, etc).

The simplicity of PyTorch is comparable to that of Keras. TF now comes with an integrated version of Keras, which is optimized for TF, so TF code can look as simple as Keras code.

In general, my impression is that, if you need to, TF provides more flexibility, but it's been a while since I used PyTorch. Furthermore, with TF 2.0, the code looks simpler, more consistent and the framework is better organized. On the other hand, PyTorch may still be easier to learn (because it doesn't have the past of TF, i.e. there are still many TF code that uses sessions, etc.).

Both TF and PyTorch are supported by big companies (respectively, Google and Facebook), but TF is more mature and there are functionalities/libraries that exist for TF that still do not exist for PyTorch (AFAIK). I doubt that there is a functionality for PyTorch that doesn't also exist for TF or Keras.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Is TF more mature though in terms of customization? Its a massive pain in TF to access variables during runtime and other dynamic variables you might want to change or set as per your wish in-between runtime depending on a flag or some event. (My experience is with TF1.9) $\endgroup$ – DuttaA Feb 12 at 16:54
  • $\begingroup$ @DuttaA In TF 2, I don't remember having had the need for accessing any variable (and I've been using it for a few months for a project). $\endgroup$ – nbro Feb 12 at 16:55
  • $\begingroup$ I required to implement something in which I needed to remove nodes between runtime, at certain periodic epochs, which I did by nulling some matrix weights. I have no idea how this would be achieved in TF. I had seen some quite complex processes to achieve that though. $\endgroup$ – DuttaA Feb 12 at 16:57
  • $\begingroup$ @DuttaA If you explain better why you needed to "remove nodes at runtime" (what exactly was your problem?), I could understand better how easily this can be done in TF 2. But it looks like you used a mask, which is quite common in machine learning. $\endgroup$ – nbro Feb 12 at 16:59
  • $\begingroup$ Yes a mask. How does it work in tf2, if I wanted to update mask periodically based on a certain criteria which is dependent on values of middle layers? I don't really need the info anymore though, but I think it would be incredibly complex in tf1.9 at least. $\endgroup$ – DuttaA Feb 12 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.