14
votes
Should neural nets be deeper the more complex the learning problem is?
Deeper models can have advantages (in certain cases)
Most people will answer "yes" to your question, see e.g. Why are neural networks becoming deeper, but not wider? and Why do deep neural networks ...
- 37.1k
11
votes
Accepted
How can I automate the choice of the architecture of a neural network for an arbitrary problem?
I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies).
The ...
- 226
7
votes
Accepted
Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?
Why has this merge proven beneficial?
If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top ...
- 396
7
votes
Accepted
Are there any learning algorithms as powerful as "deep" architectures?
Have you read the book The Master Algorithm: by Pedro Domingos?
He discusses the present day machine learning algorithms... Their strengths, weaknesses and applications...
Deep Neural Network
...
- 554
5
votes
How can I automate the choice of the architecture of a neural network for an arbitrary problem?
The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short ...
- 7,176
5
votes
Accepted
How can we get a differentiable neural network to count things?
Estimating from an observation is a function, but "really counting" is a process. Feed-forward neural networks can learn arbitrary functions from training examples, but they cannot represent ...
- 26.6k
4
votes
How to create an AI to solve a word search?
This sounds like a problem that might be solvable with a LSTM-DQN approach, as described in Language Understanding for Text-based Games using Deep
Reinforcement Learning by Narasimhan et al., 2015, ...
- 9,037
4
votes
Are there well-established ways of mixing different inputs (e.g. image and numbers)?
A more efficient way would be creating a multi input model, with something like this:
...
- 1,725
4
votes
Accepted
Are there ways to learn and practice Deep Learning without downloading and installing anything?
As I understand, I think you wish to directly try out some deep learning stuff and things like library downloading, tools downloading, and managing all these really stop you from even starting to try ...
- 142
4
votes
Are there any learning algorithms as powerful as "deep" architectures?
Deep learning is actually pretty useful (relative to other techniques) precisely when there is no simple mapping between input and output, and features from the raw input need to be aggregated and ...
- 7,176
4
votes
Accepted
What linear rectifier is better?
I've read all the papers about PReLU, LeakyReLU (...) and all the claims how it improves this and that but the little dirty secret is: most of the time it doesn't matter at all and you can't go much ...
- 341
4
votes
Accepted
Are Neural Net architectures accidental discoveries?
Although there is a strong element of "try and see" that has driven successful architectures, the drivers for what to try are often inspired by underlying theory or knowledge from other disciplines.
...
- 26.6k
4
votes
Accepted
Why do Transformers have a sequence limit at inference time?
Transformer models have limited sequence length at inference time
because of positional embeddings. But there are workarounds.
Self-attention in transformer does not distinguish the order of keys/...
- 76
4
votes
Accepted
How do I design the network for Deep Q-Network?
What is the strategy to get to a better network?
There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
- 26.6k
3
votes
Are Modular Neural Networks more effective than large, monolithic networks at any tasks?
There is indeed an investigation in progress, regarding this topic. A first publication from last march noted that modularity has been done, although not explicitly, since some time ago, but somehow ...
- 501
3
votes
Accepted
How to properly use Flatten layer?
The Flatten layer is used for collapsing an ND tensor into a 1D tensor. In your case, the inputs appear to be $28\times28$ images, so Flatten will convert that into a tensor with shape $1\times768$. ...
- 391
2
votes
How to compute the output of a neural network produced by NEAT?
The networks in NEAT are still implicitly layered. There are neurons that need to be evaluated before other neurons can be evaluated and so this gives us our layers.
If you don't know the structure ...
- 587
2
votes
What type of reinforcement learning can I do restricted to ~200MB on an average smartphone?
I skimmed through your question and understood that the state/action space is finite, so in this case, RL would be a good option for storage.
The most basic RL technique will keep track of a matrix Q ...
- 251
2
votes
Accepted
How to teach a model-based reflex agent for doing some task using machine learning methods?
The question is really broad---as stated by @thecomplexitytheorist---so difficult to give a meaningful answer. The following is about a clarification about the problem, and some directions.
A model-...
- 1,490
2
votes
Are artificial networks based on the perceptron design inherently limiting?
In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function
$$f(x) = g(w^{\top} x),$$
where $x$ ...
- 9,804
2
votes
Are Modular Neural Networks more effective than large, monolithic networks at any tasks?
A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the ...
- 7,385
2
votes
Accepted
How do neural network topologies affect GPU/TPU acceleration?
The topology of a neural network can have a significant impact on the performance of GPU and TPU acceleration.
The most ...
- 1,004
2
votes
What is the input to AlphaGo's neural network?
The input to the neural network is a $19 × 19 × 17$ image stack
comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the
presence of the current ...
- 2,018
2
votes
Should neural nets be deeper the more complex the learning problem is?
Deeper networks have more learning capacity in the sense that they can fit to more complex data. But at the same time, they are also more prone to overfitting the training data and therefore fails to ...
2
votes
Should neural nets be deeper the more complex the learning problem is?
My experience from a tactical standpoint is to start out with a smaller simple model first. Train the model and observe the training accuracy and validation loss and validation accuracy. My ...
- 694
2
votes
Accepted
Is the size of a neural network directly linked with an increase in its inteligence?
First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
All they do is trying to approximate a mathematical function with a certain degree of generalization (...
- 755
2
votes
Why does a neuron in a multi-layer network need several input connections?
There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and ...
- 1,316
2
votes
Accepted
Are there deep neural networks that have inputs connected with deeper hidden layers?
This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: ResNet, SkipRNN. In addition here you can find a paper ...
- 1,108
2
votes
Accepted
What is a unified neural network model?
A unified neural network model consists of one neural network as opposed to other models that rely on two or more neural networks.
For example, from page two of the YOLO paper:
2. Unified Detection
...
- 1,698
2
votes
Propagating gradients through an "Item Selector" network
As soon as you discretize the selection, i.e. make a hard selection (argmax) instead of a soft selection (softmax), you have a biased gradient. This is because the things you didn't select are not ...
- 1,327
Only top scored, non community-wiki answers of a minimum length are eligible