# Tag Info

65

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned. Visualizing the activations and layer weights Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out ...

61

I think this is a fairly common misconception about AI and computers, especially among laypeople. There are several things to unpack here. Let's suppose that there's something special about infinity (or about continuous concepts) that makes them especially difficult for AI. For this to be true, it must both be the case that humans can understand these ...

51

Three possibilities come to mind. The easiest is the zero-padding. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts. Recurrent NNs (RNN) are a very natural NN to choose if you have texts of ...

33

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as ...

33

Keras is a simple and high-level neural networks library, written in Python, that works as a wrapper for Tensorflow and Theano. It's easy to learn and use. Using Keras is like working with Lego blocks. It was built so that people can do quick experiments and proofs-of-concept before launching into a full-scale build process. With that in mind, it was made ...

30

It depends on what you mean by "know what is happening". Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $1/1+e^{1/1+e^{\dots}}$). However, if by 'know' you mean predicting the ...

28

Here's a snippet from an article by Gary Marcus In particular, they showed that standard deep learning nets often fall apart when confronted with common stimuli rotated in three dimensional space into unusual positions, like the top right corner of this figure, in which a schoolbus is mistaken for a snowplow: . . . Mistaking an ...

26

The bottleneck in a neural network is just a layer with less neurons then the layer below or above it. Having such a layer encourages the network to compress feature representations to best fit in the available space, in order to get the best loss during training. In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the ...

24

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature. For advanced researchers, YES. You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead ...

21

Almost all of the functionalities provided by the non-linear activation functions are given by other answers. Let me sum them up: First, what does non-linearity mean? It means something (a function in this case) which is not linear with respect to a given variable/variables i.e. $f(c1.x1 + c2.x2...cn.xn + b) != c1.f(x1) + c2.f(x2) ... cn.f(xn) + f(b).$ NOTE:...

21

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (3x3x3=27 values). So, three different 2D filters of size 3x3 can be concatenated to form this one 3D filter of size 3x3x3. The 3x3x3 RGB chunk from the picture is ...

19

Short answer is no. Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it... ...

19

How do I handle such large image sizes without downsampling? I assume that by downsampling you mean scaling down the input before passing it into CNN. Convolutional layer allows to downsample the image within a network, by picking a large stride, which is going to save resources for the next layers. In fact, that's what it has to do, otherwise your model ...

19

In theory, most neural networks can approximate any continuous function on compact subsets of $\mathbb{R}^n$, provided that the activation functions satisfy certain mild conditions. This is known as the universal approximation theorem (UAT), but that should not be called universal, given that there are a lot more discontinuous functions than continuous ones,...

18

I think your premise is flawed. You seem to assume that to "understand"(*) infinities requires infinite processing capacity, and imply that humans have just that, since you present them as the opposite to limited, finite computers. But humans also have finite processing capacity. We are beings built of a finite number of elementary particles, forming a ...

17

Deep learning is a specific variety of a specific type of machine learning. So it's possible to learn about deep learning without learning all of machine learning, but it requires learning some machine learning (because it is some machine learning). Machine learning refers to any technique that focuses on teaching the machine how it can learn statistical ...

17

To complete the first answer that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds. Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper ...

16

Others already mentioned: zero padding RNN recursive NN so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an excellent book which backs up this approach: Consider a collection of images, where each image has a different width and height. It is unclear how to model such inputs with a ...

16

David Nolen (contributor of Clojure and ClojureScript; creator of Core Logic a port of miniKanren) in a talk called LISP as too powerful stated that back in his days LISP was decades ahead of other programming languages. There are number of reasons why the language wasn't able to maintain its initial reputation. This article highlights some key points why ...

16

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random. When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course ...

15

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels? The former. In fact there is a separate kernel defined for each input channel / output channel combination. Typically for a CNN architecture, in a single filter as described by your number_of_filters parameter, ...

15

In our deep learning lecture, we discussed the following example (from Unmasking Clever Hans predictors and assessing what machines really learn (2019) by Lapuschkin et al.). Here the neural network learned a wrong way to identify a picture, i.E by identifying the wrong "relevant components". In the sensitivity maps next to the pictures, we can see that the ...

15

The paper's authors needed to implement their models anyway in order to conduct their experimentations, so why not publish the implementation? Some papers and authors actually provide a link to their own implementation, but most of the papers (that I have read) don't provide it, although some third-party implementations may already be available on Github (...

14

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and ...

13

It's all about Return On Investment. If DL is "worth doing", it's not overkill. If the cost of using DL (computer cycles, storage, training time) is acceptable, and the data available to train it is plentiful, and if the marginal advantage over alternative algorithms is valuable, then DL is a win. But, as you suggest, if your problem is amenable to ...

13

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I would quite like to see a reference to this suggestion, in case it has been misunderstood. As far as I know, there is no such effect in normal neural networks. In ...

12

A deep neural network (DNN) is nothing but a neural network which has multiple layers, where multiple can be subjective. IMHO, any network which has 6 or 7 or more layers is considered deep. So, the above would form a very basic definition of a deep network.

12

Deep learning is powerful but it is not a superior method than bayesian. They work well in what they are designed to do: Use deep learning: Cost for computation is much cheaper than cost of sampling (e.g: natural language processing) If you have highly non-linear problem If you want to simplify feature engineering If you don't have prior distribution (e.g: ...

12

NOTE: I did these calculations speculatively, so some errors might have crept in. Please inform of any such errors so I can correct it. In general in any CNN the maximum time of training goes in the Back-Propagation of errors in the Fully Connected Layer (depends on the image size). Also the maximum memory is also occupied by them. Here is a slide from ...

12

That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that expression involves some random variables, which means that there may be a wide range of different values the expression may take in any concrete, single event. ...

Only top voted, non community-wiki answers of a minimum length are eligible