New answers tagged neural-networks
0
votes
How is ChatGPT able to repeat random numbers?
You seem to see the numbers as tokens. For ChatGPT, they are often multiple tokens, there are tokens for single digits, but also for two and three digits. But a long random number is usually ...
- 214
1
vote
Open-source vocal cloning (speech-to-speech neural style transfer)
Additional projects that might be of interest:
Neural Voice Cloning with a Few Samples - NeurIPS 2018 (Sercan O. Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou)
A neural voice cloning system ...
0
votes
Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
If you have a sigmoid activation in your output layer, and linearities before, you can interpret it as a logistic regression model for binary classification (if ...
0
votes
Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
You are speaking about generalized linear models (GLM)
- 149
0
votes
Why do we need to find derivative for activation function?
This question is a duplicate to this one: Why is the derivative of the activation functions in neural networks important? you should go check the answer. Very informative!
- 1
0
votes
What is the weight matrix in self-attention?
In my mind there are two weight matrices, the one you get prior to applying softmax:
$$ \alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$
the other you get after applying the softmax:
$$ \...
1
vote
Why are the initial weights of neural networks randomly initialised?
Another theory relevant I think to this question is the “lottery ticket hypothesis”:
Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ...
- 141
1
vote
Open-source vocal cloning (speech-to-speech neural style transfer)
Tensorflow code
https://github.com/phiana/speech-style-transfer-vae-gan-tensorflow
it's the implementation of a 2021 paper. Speech style transfer, voice cloning or speech-to-speech synthesis are the ...
1
vote
How is ChatGPT able to repeat random numbers?
Existing answer is great about model generalization, but I would like to add about an important inductive bias of the Transformer model architecture used for ChatGPT.
In the Transformer model ...
- 155
0
votes
Why is a bias parameter needed in neural networks?
Let's interpret each node of a layer as a transformation of sub-feature-inputs into a certainty value for the presence (or absence) of a feature.
Then, for example a dense-layer first looks at each of ...
- 101
0
votes
Higher accuracy in the test set than in the training set
Getting higher accuracy in test set than in training set means that your test set is easier to work on than training set. Your graph shows results for only one epoch. In the 1st epoch, your train ...
- 326
1
vote
Accepted
I understand a neural network is a program, or something similar, but what exactly does it look like while it's learning?
Neural networks in practice don't look like anything.
When teaching about neural networks we may draw pictures of circles connected by lines, but this is for teaching the general idea, and any useful ...
- 639
0
votes
Higher accuracy in the test set than in the training set
Your test set images are more clean than your training set images, because you applied more types of noise to the training set images, so the classifier performs better on the cleaner images in the ...
- 639
8
votes
Why is a bias parameter needed in neural networks?
No matter what you make $W_1$ and $W_2$, if $X_1$ is 0 and $X_2$ is 0 then $W_1X_1+W_2X_2$ is 0 which (in a typical classification application) means the classifier is completely unsure which class it ...
- 639
11
votes
Why is a bias parameter needed in neural networks?
Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class ...
- 2,222
7
votes
Why is a bias parameter needed in neural networks?
If you have data generated from $y = 5\,x + 3$
How do you expect the simplest neural net $y = w_1 x$ to adjust the data ?
That is why $b$ is useful.
- 79
14
votes
Why is a bias parameter needed in neural networks?
It's not strictly "needed." In fact, if you look at things like Keras, you will see that layers have a use_bias parameter, which defaults to True, but ...
- 748
1
vote
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
The purpose of the tanh and sigmoid functions in an LSTM (Long Short-Term Memory) network is to control the flow of information through the cell state, which is the "memory" of the network.
...
- 111
0
votes
Can a neural network learn how to play the snake game effectively?
Because the snake don't know to not take a feedback in ended game. This situation prevents them recognizing the reward system.
- 1
0
votes
Accepted
Is there a mathematical proof that a binary neural network can approximate any function with arbitrary accuracy?
Yes, for a broad class, they actually do, with probability 1:
[i] Wang, Yanzhi, et al. "Universal approximation property and equivalence of stochastic computing-based neural networks and binary ...
- 156
3
votes
Accepted
Can NEAT produce output which has no connection with any other node?
To answer the main question:
Can NEAT produce output which has no connection with any other node?
Yes. This is a common property of most artificial neurons, and not really to do with NEAT.
If you ...
- 26.5k
4
votes
Why don't people use nonlinear activation functions after projecting the query key value in attention?
It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.
Attention is broadly defined as a following operation ($\text{softmax}$ is ...
- 2,222
0
votes
Is data useless for a neural network if some inputs are derivatives of other inputs?
There is a difference between adding more samples to the data (rows), and adding more features (columns). In this case we are talking about more features.
If the function $f$ is trivial, feeding extra ...
- 605
0
votes
How does the NEAT speciation algorithm work?
Even if this thread is old i want to add something to your answer, regarding to the second formula. The fitness sharing mechanism works like this:
...
0
votes
How do you name your deep learning training outputs?
You could use the most important distinctions to build a folder structure. For example I experiment with multiple model architectures (resnet, mobilenet, ...) and different types of classification (...
2
votes
Accepted
Is data useless for a neural network if some inputs are derivatives of other inputs?
No it is not useless.
The relationship may not be obvious, and having the data will allow the network to learn this 𝑓 relationship.
Further, even if 𝑓 is obvious, networks are so sample inefficient ...
- 2,005
3
votes
Accepted
What is the appropriate threshold for accuracy in logistic regression?
When labels are 0 or 1, the common threshold that is applied is 0.5
However from a theory standpoint there is no ideal threshold value. For example, if the dataset contains imbalanced classes, a ...
- 2,005
0
votes
When retraining a model are you adding a layer or changing existing node values?
By default, you are changing the weights and biases of the original model.
But it depends on the training method you choose. If you do LoRA training, they add extra weights to the model.
- 315
0
votes
Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?
The hidden and carry states of an LSTM contain the current 'embedding' of the past data that has been passed through the cell. The hidden state is also taken as output at each timestep. If, as you are ...
- 808
1
vote
Accepted
Why are neural networks so data hungry?
One example or sample is basically a pair of data $(x_i, y_i)$ with $x_i$ randomly picked from x axis and $y_i$ from the piecewise function. As you can see, it doesn't provide a lot information to ...
- 126
2
votes
How do you name your deep learning training outputs?
Depending on how many instances of models you train you can do one of the following:
For when the amount of models is still somewhat manageable: Generate a settings file together with the model file ...
- 808
Top 50 recent answers are included