7
$\begingroup$

I like the enforced indentation of Python that many don't like because I hate parenthetic typing and redundant semicolons. I like the shell interface, but why do some think Python is de facto for machine learning?

Even with straight rectified linear activation, because of sheer dimensionality, simulating circuits comprised of artificial neurons places large demands upon computing resources. Processing video in a typical adversarial artificial network algorithm requires seven nested loops.

  • Adversarial pair iteration
  • Neural net layer depth
  • Sample index
  • Frame index
  • Pixel depth
  • Vertical
  • Horizontal

We call the filter for convolution a "kernel" and pawn it off to DSPs in GPUs to squeeze out performance then use a scripting language to code in.

Why wouldn't we write deep learning code like Linus Torvalds writes kernel code, with gcc -S so we can make sure the assembly language is efficient and there are almost no cache misses? From a performance point of view, one could fly to the moon and back with C before Python even broke the tree line.

In terms of ease of experimentation, C++ is plenty object oriented so that clean abstractions can be written as .hpp files to configure and govern the kernel-efficient C that does the mechanics of parameter optimization.

We type on keyboards to code and bloc, we program microwave ovens, and some of us play musical keyboards that nicely simulate pianos. We then forget it is C/C++ underneath these highly intuitive user interfaces. I frankly, don't buy the Python argument yet.

Most of us understand that Python wrappers have been created around the efficient matrix algorithms written in FORTRAN and ported to C, and that the Python constructs for ML are relatively elegant, but is that a good reason to dismiss the fact that many C++ libraries for ML that are also elegant have been developed?

$\endgroup$
  • 2
    $\begingroup$ My sense has always been Python's main advantage is that it is an interpreted language which brings a type of development flexibility/fluidity you just don't get with compiled languages. $\endgroup$ – DukeZhou Jul 19 '18 at 20:05
  • $\begingroup$ Python is easier than c and you can make more use of python than v $\endgroup$ – AIDANMAKU Jul 20 '18 at 11:55
  • $\begingroup$ Python is easy to understand. We need more minds to solve problems. With python, more people can learn and the community expand. $\endgroup$ – Guilherme IA Jul 20 '18 at 13:25
  • 1
    $\begingroup$ @DouglasDaseeco fluidity is probably a more apt term. $\endgroup$ – DukeZhou Jul 20 '18 at 16:33
  • $\begingroup$ Similar question externallly: quora.com/Why-is-C++-not-a-good-language-for-machine-learning. As of July 22, 2018, Github.com has1,657 projects in C++ for machine learning: github.com/…. Tensorflow is one of them. There are academic articles for C++ ML approaches: MLC++, DLibML, MLPack, LibDAI, Showgun, Bob, OptiML, and a few hundred more (scholar.google.com/scholar?q=machine+learning+c%2B%2B). $\endgroup$ – FauChristian Jul 22 '18 at 8:58
4
$\begingroup$

The thing you are probably looking for is:

enter image description here

Let us see this question from 2 viewpoints:

Beginner: From a viewpoint of beginner, he needs to understand how to implement a model before optimising it. He first needs to visualise the model, see the kinds of bugs that creep in, experiment with the model to gain more intuition. Certainly possible in C/C++. But is it worth it? The time the person would take to write/debug the code in C/C++ will far outweigh the time to create i in Python/MATLAB/R. So after he gains some intermediate implementation knowledge then he can move on with C/C++.

Experts: By experts I mean programming experts. They most certainly and easily use C/C++. But the main information you missed is TensorFlow which is a ML library by itself and also serves as a core for other high level ML libraries like Keras is programmed in C++ itself. Here is more info about it. So if I am not wrong TensorFlow creates a computational graph before taking in any data, and then runs that graph on the data generally completely loaded in the RAM. So no intereference of Python in between.

Also code readability is a massive problem (at-least I face it), Writing ML algorithm in C/C++ will take huge amounts of code which may become unreadable if you look at it after a week or so if not well documented. Whereas due to inbuilt python functions you can easily read the program.

Advice from Machine Learning expert Andrew Ng, use languages like C/C++ to implement your model once you have verified your model works in a higher level language like MATLAB.

Also as @FauChristian mentioned library support is impeccable, you can combine various other libraries in other fields to convert data into a ML form which will then be used in your ML model.

$\endgroup$
3
$\begingroup$

Because there is a library for Python named NumPy.

It can do extremely fast computations on n-dimensional arrays of numbers, and all kinds of scientific / machine learning / image processing etc libraries are built on top of it.

You don't do actual computation in Python for loops, that's really slow. You call Numpy matrix operations, and they are themselves written in C (and partly in Fortran if I recall correctly, not sure if that's still the case) and mercilessly optimized for speed over the years.

$\endgroup$
2
$\begingroup$

In AI (and probably many other domains as well), time spent by the human programmers tends to be significantly more valuable / expensive than time spent running a program. Of course this is not always the case (just like it's not always the case that Python is used rather than C or C++), but it is often true.

Especially in the case of research, it's extremely important to be able to iterate over ideas rapidly. We need to be able to rapidly implement new ideas, test them, probably go through a few iterations of bugfixing, test again, etc. It's quite rare that the bottleneck in this iteration of ideas is the runtime of a new idea/algorithm. The human's time spent programming tends to be a bottleneck much more often. Ideas can often be tested rapidly on small toy problems which don't take a lot of runtime anyway, or run overnight / run while the programmer is busy writing other code.

Doing all that iterating, quickly implementing new ideas etc. tends to be easier / faster in Python than in C or C++. Of course this is not necessarily true for everyone, if someone already has a lot of experience in C++ and little experience in Python, they may be able to implement new ideas more quickly in C++. This does not appear to be the case for the majority of people though. Clear advantages that Python has in terms of how quickly new ideas can be implemented include:

  • Less verbose, don't need to type as much (e.g., results = [] in Python vs std::vector<double> results; in C++ to create an empty list that we can start appending some results to in an experiment).
  • No need to worry about memory management, pointers, all that stuff, which can be doable with experience but does inevitably require attention, has a greater likelihood of leading to extra bugs, etc.
  • No need to go through compilation/build process. This can be a very big one, which is easy to forget about.
  • Much less of a hassle to have data structures containing stuff of different types (e.g. config = {'algorithm': 'RandomForest', 'n_trees': 50} in Python vs.... no idea in C++)

Another key point is that the majority of code that people in AI write is not the performance-sensitive parts. Again, may not be true for everyone, but is true especially in research settings. Most researchers don't spend the majority of their time writing code for forwards / backwards passes in a Neural Network. They spend much more time on stuff like:

  • Preprocessing data (not always easy to write easily re-usable code, significant chunks will be project-specific / dataset-specific)
  • Experiment setup (e.g. outer training loops, printing/logging results, ...)
  • Processing results, creating all kinds of fancy plots, etc.
  • Brand new (parts of) algorithms (e.g. a new variant in the list of SGD/RMSProp/ADAM/etc., likely just a couple of lines of simple code that can be plugged straight into an existing framework, with a lot more pen&paper math behind it).

Now that last point may in some cases be performance-sensitive, but still, the initial concern will not be performance. The initial concern will be; will it work at all? This can be evaluated on simpler problems or with less performant code simply by waiting a bit longer first, it's still more important to be able to implement it at all first. As mentioned in other answers, thanks to C-based frameworks like Tensorflow and numpy, these can quite often be performant even from Python though.


Finally, the existence of well-known, easy-to-use, established, open source libraries and frameworks that have stood the test of time is extremely important. In Python, we have:

  • Numpy
  • Pandas
  • Matplotlib, Seaborn
  • scikit-learn
  • Tensorflow, Pytorch
  • XGBoost

Again, many of those have parts implemented in C/C++ when performance matters too.


Note that, when I'm talking about research, this doesn't just mean "academia". AI in industry will also often have some flavour of research. People in industry are unlikely to be implementing Neural Networks from scratch over and over and over again. They're much more likely to be re-using implementations that are already efficient (e.g. Tensorflow), but be trying to apply them to new data (where they'll have to write some new boilerplate code around it, which is quick and easy in Python), or trying new architectures, or trying to visualize data and/or results, etc.

$\endgroup$
  • 3
    $\begingroup$ @DouglasDaseeco Maybe that's the case for you specifically, but more often than not it's not the case. Using a Tesla P100 from Europe/Asia on google cloud appears to be at $1.60 per hour right now. If the choice is between renting a bunch more GPUs, or hiring an extra developer because all your developers are slower due to having to work in C++ rather than Python, the extra GPUs will often be cheaper. Anyway, a part of the point of multiple answers is that that's often not even a relevant choice; performance-sensitive parts easily can (and often are) implemented in C++ (e.g. numpy, tensorflow) $\endgroup$ – Dennis Soemers Jul 20 '18 at 18:27
  • $\begingroup$ @DouglasDaseeco Right, and when put in python it allows developers to enjoy all the benefits of python without the loss in performance (or at least at a significantly reduced loss in performance). $\endgroup$ – Dennis Soemers Jul 20 '18 at 19:12
1
$\begingroup$

The primary reason for Python being preferred is overhead. C++ automatically entails greater overhead in terms of the amount of code required to do any particular thing. Artificial intelligence is difficult to understand conceptually already, which makes programming overhead a larger issue.

With C++ modules being a way to extend the Python language, there is very little reason to not use Python. It is easier to transfer knowledge of programming in C++ to Python rather than figuring out a way to program a library in C++ extensive enough to match the current body of knowledge already incorporated into Python.

$\endgroup$
  • $\begingroup$ The "C/C++ modules" that I am referring to are part of Python extensibility rather than the C/C++ language. Where and when those features arose previous to Python, that may be a little off topic concerning the original question "Why Python, not C?" which concerns machine learning in Python versus C++ rather than the history thereof. If you are adding to the "Can it be done?" argument, I will admit that certainly it can. My argument was more of a "Should it be done?" argument, which points to the fact that it may take greater programming effort to do the same thing in C/C++ to Python. $\endgroup$ – Nathan Eggers Techno Tech Blog Jul 29 '18 at 18:11
1
$\begingroup$

Python is popular for rapid prototyping for several reasons.

  • It is a platform with a shell like Mathematica and MATLAB
  • It has built in matrix and other mathematical types
  • It is free, unlike those proprietary platforms
  • It's libraries in the ML space is currently more mature than SciLab
$\endgroup$
  • $\begingroup$ Possibly also a factor: Python is now the most widely taught introductory language (see, e.g. cacm.acm.org/blogs/blog-cacm/…). Well that's probably for the reasons you indicate here, it also helps drive the creation and maintenance of ML libraries. $\endgroup$ – John Doucette Aug 8 '18 at 0:31
  • $\begingroup$ @JohnDoucette, I think it is good that it is widely taught as an introductory language because Pylint enforces indentation. I can't even look at other people's Java or C++ code without running a code formatter on it. $\endgroup$ – FauChristian Aug 15 '18 at 11:12
1
$\begingroup$

Because, while C/C++ are not dead, they're hard to use and hard to learn. I used to like them, but now, nobody would make me learn a thing with features like undefined behavior. No way! Memory management? Why should I? I just don't want to waste my time on such stuff... and I guess, that's the common attitude among students.

There'll be always people loving "the old good low level C" and that's a good thing as we need kernels and drivers and such. It has its place.

C++ is higher level, but it's become much too complicated for most of us, who prefer to spend learning time on algorithms rather than on the language itself.

Everything what can be reasonably done with less effort, should not be done with more effort. I was using C++ for years (some long time ago), and spend only a few hours with Python, but I'd always choose the latter for a few thousand lines project because of the efficiency of coding.

The efficiency of execution may be improved later, possibly even by rewriting a small part in C++. Oftentimes, more can be gained by implementing a smarter algorithm.

$\endgroup$
  • $\begingroup$ I'm not sure they are hard to learn. My first UNIX program was hello.c from Kernighan & Ritchie. Then I scanned the rest of the book in an hour and wrote a suite of analysis programs by the end of that week. I learned C++ by reading Stroustrup more carefully in 3 work days. It was then easy to write a simulation program before the end of the fourth day. ~~ It may be that Python is abstracted like MATLAB but doesn't require my credit card. ~~ Plus the silicon valley Pythonians walk around the Moscone Center every event with really rad t-shirts. $\endgroup$ – FauChristian Aug 15 '18 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.