Many authors of research papers in AI (e.g. arXiv) write their neural networks from the ground-up, using low-level languages like C++ to implement their theories. Can existing open source frameworks also be used for this purpose, or are their implementations too limited?

Can, for example, TensorFlow be used to craft an original network architecture that shows improvements on existing benchmarks? Can original mathematical work be coded into a high-level framework like TensorFlow such that original research on network architectures/approaches be demonstrated in a paper?

A quick search reveals many papers using C++ in their implementation:

  • 1
    $\begingroup$ Hello. Welcome to AI SE! When you say "Many authors of research papers in AI (e.g. arXiv) write their neural networks from the ground-up, using low-level languages like C++ to implement their theories", can you please provide 2-3 examples (links to papers or official implementations) where that has been the case? It seems to me that most AI researchers use frameworks (such as PyTorch and TensorFlow) to implement their models, so I don't think that sentence is true, that's why I'm asking for examples. $\endgroup$ – nbro Jan 5 at 22:38
  • 2
    $\begingroup$ Literally nobody does this, the one exception maybe is YOLO where the guy wrote a C based library. $\endgroup$ – FourierFlux Jan 6 at 0:56
  • $\begingroup$ @FourierFlux Literally nobody does this? A 3 second Google search reveals 5 papers off the top. I didn’t say most authors, I said many, which is true. $\endgroup$ – Cybernetic Jan 6 at 1:17
  • 3
    $\begingroup$ Did you read the papers you referenced? Except for the spiking neuron one all of the papers were not all focused on developing AI algorithms, but the implementation on specific hardware. Basically nobody who is interested in developing AI algorithms spends their time developing toolkits. $\endgroup$ – FourierFlux Jan 6 at 1:55
  • 1
    $\begingroup$ Look up the papers from top ML/AI conferences such as ICML, ICLR, AAAI, NeurIPS etc. All papers with an experimental content I've read used either Tensorflow, or PyTorch to do their experiments. $\endgroup$ – SpiderRico Jan 6 at 2:53

Your statement that researchers build their network from the ground-up using C++ or some other low level library couldn't be further from the truth.

You could take a look at this analysis showing the popularity of these two frameworks in the top ML conferences. The following Figure is taken from there.

In CVPR-2020, for example, TensorFlow and pytorch combined for over 500 papers! Furthermore, because the two most active research entities (Google and Facebook) are backing these two frameworks, they are used in some of the most impactful research studies.

I want to give some reasons that support the popularity of these frameworks, but first I'm going to rephrase your question a bit:

Why use TensorFlow/Pytorch in python rather than build your model on your own using C++?

Note: The reason I rephrased the question is because TensorFlow and PyTorch both have a C++ APIs.

Why are these frameworks so popular in contrast to lower-level programming languages?

Some reasons are the following

  • Rapid prototyping. Languages link C++, have bloated syntaxes, require low-level operations (e.g. memory management) and cannot be run interactively. This means it takes someone much less time to create and test a model in python than it does in C++.

  • No need to re-invent the wheel. Some operations are common in most networks (e.g. backpropagation), why re-implement them? Other functionalities are hard to implement on your own (e.g. parallel processing, GPU computation). Do data scientists need to have such a strong technical background to research neural networks?

  • Open-source. They benefit from being opensource and can offer a great deal of tools at your disposal for building neural networks. You want to add batchnorm to your network? No worries, just import it and add it in a single line! Also, they offer the perfect opportunity for sharing pretrained models.

  • They are optimized. These frameworks are optimized to run as fast as possible on GPUs (if available) or CPUs. It would be virtually impossible for someone to write code that runs as fast on his own.

  • $\begingroup$ Your comment “couldn’t be further from the truth” makes no sense. I said many, not most. This is confirmed from a casual Google search. Don’t exaggerate for effect. Read the question. $\endgroup$ – Cybernetic Jan 6 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.