71

Introduction The term self-supervised learning (SSL) has been used (sometimes differently) in different contexts and fields, such as representation learning [1], neural networks, robotics [2], natural language processing, and reinforcement learning. In all cases, the basic idea is to automatically generate some kind of supervisory signal to solve some task (...


62

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general ...


58

Machine learning has been defined by many people in multiple (often similar) ways [1, 2]. One definition says that machine learning (ML) is the field of study that gives computers the ability to learn without being explicitly programmed. Given the above definition, we might say that machine learning is geared towards problems for which we have (lots of) data ...


51

Three possibilities come to mind. The easiest is the zero-padding. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts. Recurrent NNs (RNN) are a very natural NN to choose if you have texts of ...


44

Python comes with a huge amount of inbuilt libraries. Many of the libraries are for Artificial Intelligence and Machine Learning. Some of the libraries are TensorFlow (which is a high-level neural network library), scikit-learn (for data mining, data analysis and machine learning), pylearn2 (more flexible than scikit-learn), etc. The list keeps going and ...


33

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as ...


31

The difference is mostly in the number of layers. For a long time, it was believed that "1-2 hidden layers are enough for most tasks" and it was impractical to use more than that, because training neural networks can be very computationally demanding. Nowadays, computers are capable of much more, so people have started to use networks with more layers and ...


31

Practically all of the most popular and widely used deep-learning frameworks are implemented in Python on the surface and C/C++ under the hood. I think the main reason is that Python is widely used in scientific and research communities, because it's easy to experiment with new ideas and code prototypes quickly in a language with minimal syntax like Python. ...


31

By "company A has a large human face database so that it can train its facial recognition program more efficiently" the article probably means that there is a training dataset $S$ of the form $$ S = \{ (\mathbf{x}_1, y_1), \dots,(\mathbf{x}_N, y_N) \} $$ where $\mathbf{x}_i$ is an image of the face of the $i$th human and $y_i$ (which is often called a ...


26

You don't need a powerful language for programming AI. Most of the developers are using libraries like Keras, Torch, Caffe, Watson, TensorFlow, etc. Those low level libraries are highly optimized and handle all the tough work. They are built with high-performance languages, like C, C++. Python is just there for high level task like describing the neural ...


22

I believe it would be more accurate to say that (some) search engines use AI. Broadly saying "search engines are AI" is not really correct. At the core, most search engines are nothing more than an inverted text index using something like tf–idf scoring. That's a very mechanical/simple thing that nobody would really call AI. But more sophisticated search ...


22

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and ...


22

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them. Transformers Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...


21

Machine learning is a subset of artificial intelligence. Roughly speaking, it corresponds to its learning side. There is no "official" definitions, boundaries are a bit fuzzy.


21

AI is vulnerable from two security perspectives the way I see it: The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with. ...


21

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless). In practice what you want ...


20

I haven't seen an answer from a trusted source, but I'll try to answer this myself, with a simple example (with my current knowledge). In general, note that training a MLP using back-propagation is usually implemented with matrices. Time complexity of matrix multiplication The time complexity of matrix multiplication for $M_{ij} * M_{jk}$ is simply $\...


17

Deep learning is a specific variety of a specific type of machine learning. So it's possible to learn about deep learning without learning all of machine learning, but it requires learning some machine learning (because it is some machine learning). Machine learning refers to any technique that focuses on teaching the machine how it can learn statistical ...


16

Artificial intelligence According to the book Artificial Intelligence: A Modern Approach (section 1.1), artificial intelligence (AI) has been defined in multiple ways, which can be organized into 4 categories. Thinking Humanly Thinking Rationally Acting Humanly Acting Rationally Figure 1.1 (of the same book) contains 8 definitions (by renowned people like ...


16

Others already mentioned: zero padding RNN recursive NN so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an excellent book which backs up this approach: Consider a collection of images, where each image has a different width and height. It is unclear how to model such inputs with a ...


16

David Nolen (contributor of Clojure and ClojureScript; creator of Core Logic a port of miniKanren) in a talk called LISP as too powerful stated that back in his days LISP was decades ahead of other programming languages. There are number of reasons why the language wasn't able to maintain its initial reputation. This article highlights some key points why ...


16

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random. When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course ...


15

There are some wonderful resources for keeping up to date in the ML community. Here are just a handful that a coworker showed me: Deep Learning Monitor: this site contains hot and new papers along with tweets that are popularized by the community! You can even checkout RL papers specifically here arxiv-sanity: this site updates with popular and new papers ...


15

AIXI is a Bayesian, non-Markov, reinforcement learning and artificial general intelligence agent that is incomputable, given the involved incomputable Kolmogorov complexity. However, there are approximations of AIXI, such as AIXItl, described in Universal Artificial Intelligence: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic ...


14

The general research area is known as grammar induction. It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding parse tree. The training set often consists of both positive and negative examples. There is no single best method for achieving this, but some of the techniques ...


14

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron. It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards ...


14

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise). In Reinforcement Learning, ...


14

State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask when you might prefer to try an SVM. SVMs have several advantages: Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear ...


13

It's all about Return On Investment. If DL is "worth doing", it's not overkill. If the cost of using DL (computer cycles, storage, training time) is acceptable, and the data available to train it is plentiful, and if the marginal advantage over alternative algorithms is valuable, then DL is a win. But, as you suggest, if your problem is amenable to ...


13

When it comes to classification problem in machine learning, the cross entropy and KL divergence are equal. As already stated in the question, the general formula is this: $$H(p, q) = H(p) + D_{KL}(p||q)$$ Where $p$ a “true” distribution and $q$ is an estimated distribution, $H(p, q)$ is the cross-entropy, $H(p)$ is the entropy and $D$ is the Kullback-...


Only top voted, non community-wiki answers of a minimum length are eligible