We just launched a new product: Collectives on Stack Overflow. How do they work? Listen to learn more.

Hot answers tagged

14

Some back of the envelope calculations : number of neurons in AI systems The number of neurons in AI systems is a little tricky to calculate, Neural Networks and Deep Learning are 2 current AI systems as you call them, specifics are hard to come by (If someone has them please share), but data on parameters do exist, parameters are more analogous to ...


9

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is $$o_i^l = \sigma(\mathbf{x}_i^l \cdot \mathbf{w}_i^l + b_i^l),$$ where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity ...


7

Soon enough but that doesn't mean anything at all. In machine learning the word neuron represents a calculation whereas in brain the word neuron represent a specific type of cell which is a biochemical system.


6

Neural networks don't model biological neurons. They are at best inspired by biological neurons, in that they get excited by certain inputs and fire once the excitation crosses a threshold. And this second point even holds only approximately because the backpropagation algorithm needs smoothed out steps to learn by gradient descent. And backpropagation is ...


5

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network. By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with ...


5

One probable hardware limiting factor is internal bandwidth. A human brain has $10^{15}$ synapses. Even if each is only exchanging a few bits of information per second, that's on the order of $10^{15}$ bytes/sec internal bandwidth. A fast GPU (like those used to train neural networks) might approach $10^{11}$ bytes/sec of internal bandwidth. You could ...


4

It looks like you really have two questions here. I'll try to answer the first one, and you should think about making a separate question for the second. There is research into using simulated models of biologically realistic neurons. While there are large projects like the Human Brain Project aimed at simulating human brains, there is also a lot of lower-...


4

No, here is why. No approach can simulate the mind with 100% accuracy. a major notion that AI theorist refuse to note is that you cant take an orange and by virtue of technology turn it into an apple lets apply the same logic here. neurons are temporary things in our brains, daily we are trimming our brains and growing our brains, in order to "Engineer" a ...


4

I assume you're talking about a perceptron threshold function. One definition of it with an explicit threshold is $$f(\textbf{x})= \begin{cases} 1& \text{if } \textbf{w}\cdot\textbf{x} > t\\ 0& \text{otherwise} \end{cases}.$$ Another form with a bias is $$f(\textbf{x})= \begin{cases} 1& \text{if } \textbf{w}\cdot\textbf{x} + b > 0\\ 0&...


4

The basic calculation for a single neuron is of the form $$\sigma\left(\sum_{i} x_i w_i \right),$$ where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be $$c\,a_c ...


4

In short I mentioned in another post, how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of central nervous system development and synaptogenesis. A bit more detail (with ...


4

State of Rosehip Research The Rosehip neuron is an important discovery, with vast implications to AI and its relationship to the dominant intelligence on earth for at least the last 50,000 years. The paper that has spawned other articles is Transcriptomic and morphophysiological evidence for a specialized human cortical GABAergic cell type, Buldog et. al.,...


4

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of nonlinear function to its input. There are lots of different variations, but in the end the combination of the nonlinear ...


3

It is true that the current Machine learning is based on treating neurons as a component in the whole complexity , mesh of neurons. The focus is more on the architecture rather than understanding or imitating the basic block of it more clearly , i.e. the neurons. Anirban Bandhopadhyay is a biologist and Neurologist who has studied how the harmony changes ...


3

The answers so far haven't answered the question numerically, so here is my attempt to steer them in the direction I was seeking: The freely available Deep Learning Book has the following figure on page 27: I question the blue fit line, as it seems that data points may be better described by a parabolic or exponential function. In any case, based upon ...


3

Well, adding gaussian noise is a very common regularisation method. Maybe this paper is interesting to you. They also have very small datasets. In the end there is only so much you can get out of a given dataset.


3

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you are increasing the number of discernible areas. As you increase the number of lines you can use to break up the input space, the lines can be placed to ...


3

While interesting, this is all rendered somewhat moot if you think about what will happen once we understand how the brain works. After all, once we understood flight, we didn't start making birds. The same goes for AI. Here are just a few ways in which human brains and digital brains can't be compared. The digital brain won't have to worry about food and ...


3

Good question. It is related to the genetic algorithm concept, automated bug detection, and continuous integration. Early Genetically Inspired Algorithms Some of the Cambridge LISP code in the 1990s worked deliberately toward self-improvement, which is not the same as self-repair, but the two are conceptual siblings. Some of those early LISP algorithms ...


3

Yes, this was an active area of research in a number of different AI fields. Probably the most directly related work is Bongard, Zykov & Lipson's self-repairing robots from the early 2000's. There's some more recent work from Mark Yim that you can see here too. There are lots of different ways to do this, but Bongard et al's approach was probably the ...


3

Principles of Computational Modelling in Neuroscience by David Sterratt, Bruce Graham, Andrew Gillies and David Willshaw discuss it in Chapter 7 (The synapse) and also in Chapter 8 (Simplified models of neurons). Especially in chapter 8, they discuss how to add excitatory or inhibitory synapses to integrate and fire neuron. There are various ways to add ...


3

Taking the question from comments on nbro's answer. Am I wrong to see a clear relationship between how we are currently training networks and the classic function that defines a line? You are right about it. This is an intuitive way to understand neural networks. You can create a neural network that only does simple linear regression, by using linear ...


3

In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, electrical engineering, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits. There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1],...


2

In biology, when the presynaptic releases a neurotransmitter (a positive amount of them, obviously), this neurotransmitter reaches the postsynaptic vesicles causing an excitatory (depolarization) or inhibitory (hyperpolarization) effect, depending on the kind of postsynaptic vesicle in next cell dendrites. If the total amount of depolarization (all dendrites)...


2

This has been my field of research. I've seen the previous answers that suggest that we don't have sufficient computational power, but this is not entirely true. The computational estimate for the human brain ranges from 10 petaFLOPS ($1 \times 10^{16}$) to 1 exaFLOPS ($1 \times 10^{18}$). Let's use the most conservative number. The TaihuLight can do 90 ...


2

I recommend preprocessing images and feeding pixel values of several combined images. Some ideas: Preprocess all images to grayscale if possible. It’ll reduce the number of input neurons. (As long as this step doesn’t introduce large overhead) Select some $\gamma$ value such that 0 < $\gamma$ < 1. Generate (ie. Select from your game) $n$ sequential ...


2

The most generic approach is to input all pixels as you have suggested. A CNN would be the best architecture for that. To provide information like speed or velocity, you can feed more than one frame to the CNN (e.g. the last 5 frames or whatever provides enough information). The CNN can learn movement information by comparing those images. If you want to ...


2

There is indeed an investigation in progress, regarding this topic. A first publication from last march noted that modularity has been done, although not explicitly, since some time ago, but somehow training keeps being monolithic. This paper assess some primary questions about the matter and compares training times and performances on modular and heavily ...


2

A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. ...


2

Yes, for many sensory inputs there is indeed something similar to normalization. But its not rally the same as in classical data analytics compared to what eg min/max normalization does or other technics. Lets look on some examples and considerations: mammals don't perceive heat or loudness in a linear way. This is because already many sensory receptors ...


Only top voted, non community-wiki answers of a minimum length are eligible