# Tag Info

59

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned. Visualizing the activations and layer weights Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out ...

42

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general ...

40

Three possibilities come to mind. The easiest is the zero-padding. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts. Recurrent NNs (RNN) are a very natural NN to choose if you have texts of ...

32

It appears to not be published yet; the best available online are these slides for this talk. (Several people reference an earlier talk with this link, but sadly it's broken at time of writing this answer.) My impression is that it's an attempt to formalize and abstract the creation of subnetworks inside a neural network. That is, if you look at a standard ...

32

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as ...

29

It depends on what you mean by "know what is happening". Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $1/1+e^{1/1+e^{\dots}}$). However, if by 'know' you mean predicting the ...

28

The difference is mostly in the number of layers. For a long time, it was believed that "1-2 hidden layers are enough for most tasks" and it was impractical to use more than that, because training neural networks can be very computationally demanding. Nowadays, computers are capable of much more, so people have started to use networks with more layers and ...

23

TLDR: The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting). See ...

22

Yes, it has been done! However, the applications aren't to replace calculators or anything like that. The lab I'm associated with develops neural network models of equational reasoning to better understand how humans might solve these problems. This is a part of the field known as Mathematical Cognition. Unfortunately, our website isn't terribly informative,...

20

AI is vulnerable from two security perspectives the way I see it: The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with. ...

19

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature. For advanced researchers, YES. You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead ...

16

David Nolen (contributor to Clojure and ClojureScript; creator of Core Logic a port of miniKanren) in a talk called LISP as too powerful stated that back in his days LISP was decades ahead of other programming languages. There are number of reasons why the language wasn't able to maintain it's name. This article highlights som key points why LISP is good ...

16

You don't need a powerful language for programming AI. Most of the developers are using libraries like Keras, Torch, Caffe, Watson, TensorFlow, etc. Those libraries are highly optimized and handle all the though work, they are built with high performance languages, like C. Python is just there to describe the neural network layers, load data, launch the ...

16

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless). In practice what you want ...

15

Your question is quite broad, but here are some tips: For feedforward networks, see this question: @doug's answer has worked for me. There's one additional rule of thumb that helps for supervised learning problems. The upper bound on the number of hidden neurons that won't result in over-fitting is: $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$...

14

Now the question is how to handle such large image sizes where there is no privileges of downsampling I assume that by downsampling you mean scaling down the input before passing it into CNN. Convolutional layer allows to downsample the image within a network, by picking a large stride, which is going to save resources for the next layers. In fact, that's ...

13

"Backprop" is the same as "backpropagation": it's just a shorter way to say it. It is sometimes abbreviated as "BP".

13

To supplement the previous answer: there is a paper on this that is mostly about learning low-level capsules from raw data, but explains Hinton's conception of a capsule in its introductory section: http://www.cs.toronto.edu/~fritz/absps/transauto6.pdf It's also worth noting that the link to the MIT talk in the answer above seems to be working again. ...

13

Short answer is no. Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it... ...

13

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random. When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course ...

13

Almost all of the functionalities provided by the non-linear activation functions are given by other answers. Let me sum them up: First, what does non-linearity mean? It means something (a function in this case) which is not linear with respect to a given variable/variables i.e. $f(c1.x1 + c2.x2...cn.xn + b) != c1.f(x1) + c2.f(x2) ... cn.f(xn) + b.$ ` What ...

13

Even if it’s impossible to answer this question properly, as non trivial is not well defined (maybe the author will edit this questions later, to specify it better), I take the opportunity to point out this paper which looks interesting to me Smallest Neural Network to Learn the Ising Criticality Assuming you have a general idea of the Ising Model I think ...

12

If by true AI, you mean 'like human beings', the answer is - no-one knows what the appropriate computational mechanisms (neural or otherwise) are or indeed whether we are capable of constructing them. What Artificial Neural Nets (ANNs) do is essentially 'nonlinear regression' - perhaps this is not a sufficiently strong model to express humanlike behaviour. ...

12

Section 4.2 of "Essentials of Metaheuristics" has a wealth of information on alternative ways of encoding graph structures via Genetic Algorithms. With particular regard to evolving ANNs, I would personally not be inclined to implement this sort of thing 'from scratch': The field of neuroevolution has been around for some time, and the implementation some ...

12

Others already mentioned: zero padding RNN recursive NN so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an excellent book which backs up this approach: Consider a collection of images, where each image has a different width and height. It is unclear how to model such inputs with a ...

12

Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict: Big area, higher price Small amount of bedrooms, lower price The second layer ...

11

I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies). The original NEAT paper involves evolving weights for connections, but if you only want a topology, you can use a weighting algorithm instead. You can also mix ...

11

Early success on prime number testing via artificial networks is presented in A Compositional Neural-network Solution to Prime-number Testing, László Egri, Thomas R. Shultz, 2006. The knowledge-based cascade-correlation (KBCC) network approach showed the most promise, although the practicality of this approach is eclipsed by other prime detection algorithms ...

11

Convolutional Neural Networks (CNNs) are neural networks with architectural constraints to reduce computational complexity and ensure translational invariance (the network interprets input patterns the same regardless of translation— in terms of image recognition: a banana is a banana regardless of where it is in the image). Convolutional Neural Networks ...

11

NOTE: I did these calculations speculatively, so some errors might have crept in. Please inform of any such errors so I can correct it. In general in any CNN the maximum time of training goes in the Back-Propagation of errors in the Fully Connected Layer (depends on the image size). Also the maximum memory is also occupied by them. Here is a slide from ...

Only top voted, non community-wiki answers of a minimum length are eligible